The current file list generator will do the following things:

• Completely (and recursively) ignore anything that isn't group readable.
• Mark things which are not world readable by appending a '-' to the file type. So you have 'd-' and 'f-' for directories and files which are 'restricted'.

This is sufficient for quick-fedora-mirror's needs. But here are some open questions:

• Is that single bit of information enough for your needs? It seems it should be; we really don't expect and shouldn't really try to handle 'weird' permissions like directories without matching 'r' and 'x', permissions, or things which are world-readable but not group-readable. And basically a file that's not accessible to any mirror shouldn't show up anywhere at all.

• Should a restricted directory permission apply recursively to everything below it? This would simplify the client a bit, and I think I'm going to go ahead and make it work that way. But this would mean that the 'restricted' flag would reflect the 'effective' permissions on the content and not the real content, and so could not be used to verify that the the permissions on a file you currently have are correct. That's not a function of quick-fedora-mirror and so it doesn't bother me, but I want to make sure it doesn't get in the way of some other use someone might have for the file list.

For the last one, I could denote things which are 'virtually unreadable' (because they're below a restricted directory) differently from things which are truly unreadable due to permissions. This would still simplify the client while providing enough information to distinguish the two cases. I'll code it up and see how it looks.

The current file list generator will do the following things:

Completely (and recursively) ignore anything that isn't group readable.
Mark things which are not world readable by appending a '-' to the file type. So you have 'd-' and 'f-' for directories and files which are 'restricted'.

Makes sense.

Should a restricted directory permission apply recursively to everything below it? This would simplify the client a bit, and I think I'm going to go ahead and make it work that way. But this would mean that the 'restricted' flag would reflect the 'effective' permissions on the content and not the real content, and so could not be used to verify that the the permissions on a file you currently have are correct. That's not a function of quick-fedora-mirror and so it doesn't bother me, but I want to make sure it doesn't get in the way of some other use someone might have for the file list.

That would be wrong from MirrorManager's point of view. MirrorManager needs it only at the top level to ignore everything below it. Especially during crawling. If all files and directories below are also marked not world-readable this means more changes in the database. Which is probably not a real problem but not really necessary.

This should all be implemented now, but the new generator is not yet deployed in infrastructure. Will talk to nirik about that tomorrow.

What's done is that you get 'f-' or 'd-' for content that's group readable but not world readable. However, in order to vastly simplify things on the client side, content which is below a 'd-' directory but which itself is world-readable is marked with '*'.

This means that if you just care about the permissions something should have, 'f', 'd', 'f' and 'd' are all equivalent and imply world-readable content. 'd-' and 'f-' imply restricted content.

If you care about what content a mirror should have, all mirrors should have 'f' and 'd' stuff, tier1 mirrors should additionally have 'f-', 'f', 'd-' and 'd'.

The end result should tell mirrormanager what it needs to know, but it also makes it really easy for the client to figure out what it should try to transfer without having to figure out if a particular file is somewhere beneath a protected directory.

Note also that links may also be 'l-' (if they point to content that's restricted) or 'l' (if the link itself is beneath a restricted directory), but a link at the top of the tree to something that's below a restricted (or even unreadable) directory isn't marked 'l'. That's doable but I don't think it's particularly important. (And to be completely fair I haven't done a lot of testing of links yet.)