geli and BIO_FLUSH and/or BIO_ORDERED issue?

[John-Mark Gurney]

I was looking at geli and I'm not sure if it's implementing BIO_FLUSH
and/or BIO_ORDERED properly...

From my understanding is the BIO_ORDERED is suppose to wait for the

previous _WRITES to complete before returning so that you can ensure
that data is on disk, i.e. _ORDERED is set on a BIO_FLUSH...

BIO_ORDERED is handled by diskq_* code such that when you add an _ORDERED
command, all commands are put after it, but there doesn't appear to
be any code to ensure that an _ORDERED command waits for prevoius
pending commands to complete..

This is extra obvious in eli in that a _FLUSH is immediately dispatched,
even when there may be _WRITEs that haven't been finished encrypting and
sent down to the disk to get _FLUSHed...

Any comments about this?

Thanks.

[Pawel Jakub Dawidek]

Post by John-Mark Gurney
I was looking at geli and I'm not sure if it's implementing BIO_FLUSH
and/or BIO_ORDERED properly...

From my understanding is the BIO_ORDERED is suppose to wait for the

previous _WRITES to complete before returning so that you can ensure
that data is on disk, i.e. _ORDERED is set on a BIO_FLUSH...
BIO_ORDERED is handled by diskq_* code such that when you add an _ORDERED
command, all commands are put after it, but there doesn't appear to
be any code to ensure that an _ORDERED command waits for prevoius
pending commands to complete..
This is extra obvious in eli in that a _FLUSH is immediately dispatched,
even when there may be _WRITEs that haven't been finished encrypting and
sent down to the disk to get _FLUSHed...
Any comments about this?

Hmm, BIO_ORDERED was introduced pretty recently and GEOM classes were
not updated to honour it, but it also seems to be to complex to handle
in GEOM classes. I wonder if we could hold off new writes and wait for
the in-progress writes in GEOM if we spot BIO_ORDERED request without
the need to implement this logic in GEOM classes.
--
Pawel Jakub Dawidek http://www.wheelsystems.com
FreeBSD committer http://www.FreeBSD.org
Am I Evil? Yes, I Am! http://tupytaj.pl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-geom/attachments/20120922/9492b65e/attachment.pgp

[John-Mark Gurney]

Post by Pawel Jakub Dawidek

Post by John-Mark Gurney
I was looking at geli and I'm not sure if it's implementing BIO_FLUSH
and/or BIO_ORDERED properly...

From my understanding is the BIO_ORDERED is suppose to wait for the

previous _WRITES to complete before returning so that you can ensure
that data is on disk, i.e. _ORDERED is set on a BIO_FLUSH...
BIO_ORDERED is handled by diskq_* code such that when you add an _ORDERED
command, all commands are put after it, but there doesn't appear to
be any code to ensure that an _ORDERED command waits for prevoius
pending commands to complete..
This is extra obvious in eli in that a _FLUSH is immediately dispatched,
even when there may be _WRITEs that haven't been finished encrypting and
sent down to the disk to get _FLUSHed...
Any comments about this?

Hmm, BIO_ORDERED was introduced pretty recently and GEOM classes were
not updated to honour it, but it also seems to be to complex to handle
in GEOM classes. I wonder if we could hold off new writes and wait for
the in-progress writes in GEOM if we spot BIO_ORDERED request without
the need to implement this logic in GEOM classes.

Yeh. When I was looking at it, it definately seems like it should be
something that we provide a generic method of handling (as part of
bioq_*), since all the geom classes need to handle it...

It'll be a bit difficult since we'd need to introduce some syncronization
between the up/down threads to start the new writes when the previous
writes finish... And with a class like geli, you can get better latency
if we move handling into the class though...

[Pawel Jakub Dawidek]

Post by John-Mark Gurney

Post by Pawel Jakub Dawidek

Post by John-Mark Gurney
I was looking at geli and I'm not sure if it's implementing BIO_FLUSH
and/or BIO_ORDERED properly...

From my understanding is the BIO_ORDERED is suppose to wait for the

previous _WRITES to complete before returning so that you can ensure
that data is on disk, i.e. _ORDERED is set on a BIO_FLUSH...
BIO_ORDERED is handled by diskq_* code such that when you add an _ORDERED
command, all commands are put after it, but there doesn't appear to
be any code to ensure that an _ORDERED command waits for prevoius
pending commands to complete..
This is extra obvious in eli in that a _FLUSH is immediately dispatched,
even when there may be _WRITEs that haven't been finished encrypting and
sent down to the disk to get _FLUSHed...
Any comments about this?

Hmm, BIO_ORDERED was introduced pretty recently and GEOM classes were
not updated to honour it, but it also seems to be to complex to handle
in GEOM classes. I wonder if we could hold off new writes and wait for
the in-progress writes in GEOM if we spot BIO_ORDERED request without
the need to implement this logic in GEOM classes.

Yeh. When I was looking at it, it definately seems like it should be
something that we provide a generic method of handling (as part of
bioq_*), since all the geom classes need to handle it...

No, in most cases this is not a problem, because most of GEOM classes
just pass all I/O requests without any reordering, so it is enough if
the very last layer (eg. disk driver) handles BIO_ORDERED properly.

I thought what you meant with GELI was that it can reorder writes, for
which it needs more time with BIO_FLUSH requests that it handles
immediately.
--
Pawel Jakub Dawidek http://www.wheelsystems.com
FreeBSD committer http://www.FreeBSD.org
Am I Evil? Yes, I Am! http://tupytaj.pl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-geom/attachments/20120923/12364b06/attachment.pgp

[John-Mark Gurney]

Post by Pawel Jakub Dawidek

Post by John-Mark Gurney

Post by Pawel Jakub Dawidek

Post by John-Mark Gurney
I was looking at geli and I'm not sure if it's implementing BIO_FLUSH
and/or BIO_ORDERED properly...

From my understanding is the BIO_ORDERED is suppose to wait for the

previous _WRITES to complete before returning so that you can ensure
that data is on disk, i.e. _ORDERED is set on a BIO_FLUSH...
BIO_ORDERED is handled by diskq_* code such that when you add an _ORDERED
command, all commands are put after it, but there doesn't appear to
be any code to ensure that an _ORDERED command waits for prevoius
pending commands to complete..
This is extra obvious in eli in that a _FLUSH is immediately dispatched,
even when there may be _WRITEs that haven't been finished encrypting and
sent down to the disk to get _FLUSHed...
Any comments about this?

Hmm, BIO_ORDERED was introduced pretty recently and GEOM classes were
not updated to honour it, but it also seems to be to complex to handle
in GEOM classes. I wonder if we could hold off new writes and wait for
the in-progress writes in GEOM if we spot BIO_ORDERED request without
the need to implement this logic in GEOM classes.

Yeh. When I was looking at it, it definately seems like it should be
something that we provide a generic method of handling (as part of
bioq_*), since all the geom classes need to handle it...

No, in most cases this is not a problem, because most of GEOM classes
just pass all I/O requests without any reordering, so it is enough if
the very last layer (eg. disk driver) handles BIO_ORDERED properly.
I thought what you meant with GELI was that it can reorder writes, for
which it needs more time with BIO_FLUSH requests that it handles
immediately.

I did mean that GELI can reorder writes, since when it schedules the
writes on a queue, there is nothing that ensures that each thread on
an SMP system (I have 6 on mine) will complete the requets in the
order they were queued..

So, even if GELI simply added the _FLUSH command to the queue, we'd
still need to have a method for GELI to serialize the writes either
going into the queue (by holding off the _ORDERED command till all
outstanding _WRITES are back, but this will increase latency) or out
of the queue (by giving each bio an ordered id, recording which id's
have the _ORDERED flag set, and only submiting the _ORDERED command
once all the previous _WRITES have been submitted)...