quick-fedora-mirror

#16 Exclusion mechanism
Closed: Fixed Opened by tibbs.

Closed: Fixed
Reopen Issue

You can add arguments to rsync to exclude files and directories. This partially works but the excluded files are going to show up as missing with every run, and a checkin won't know that they weren't transferred.

  • It's not possible to translate from an rsync exclude list to a regex or shell glob.
  • It may be possible to go from a shell glob to an exclude list, and to a regex.
  • It may also be possible to just use a regex with rsync as long as you don't use the syntax supported by --exclude-from.
  • I do not know if it's possible to use grep -v on a file list to get the same thing that rsync would transfer if passed --exclude.

So, first test the last item and if it works, then take a list of exclude patterns. Turn those into --exclude options for rsync, and also grep -v them out of the various file lists.

If not, let the admin add exclude options as they wish, but provide a regex they can grep out of the file lists and leave it up to them to get things correct. A test mirror can tell if something is missing on either side.

Or, you know, just specify a regex that we grep out of the file lists, since we never transfer anything other than what is in the transfer list. Doh.

At least this just became much easier.

Or, even easier, allow the specification of a list of arches to exclude, and then do whatever is necessary to exclude them, And also an option to exclude iso files.

So, I really want this, because I keep an extremely incomplete mirror (no i686, no ARM, no SRPMs, no images, some other exclusions). I really need to stop the script from adding all the stuff I exclude to the file list every time.

Here's a very simple way we could do it:

diff --git a/quick-fedora-mirror b/quick-fedora-mirror
index 3076c6f..7f1a64b 100755
--- a/quick-fedora-mirror
+++ b/quick-fedora-mirror
@@ -76,6 +76,13 @@ finish () {
     exit 0
 }
+# Client-side file list filtering.
+filter () {
+    if [[ -n $FILTEREXP ]]; then
+        sed -i -r -e "$FILTEREXP" $1
+    fi
+}
+
 do_rsync () {
     # Options: source, destination.
     # The name of an array containing additional options.
@@ -462,6 +469,11 @@ for module in $MODULES; do
                  {if (s && \$2 == \"f\" || \$2 == \"l\")
                         print \"$moduledir/\" \$4 \"\t\" \$3}
                 " $fl > allfilesizes-$module
+
+    # Filter the list if requested.
+    filter allfilesizes-$module
+
+    # Produce the file list.
     awk -F '\t' '{print $1}' allfilesizes-$module > allfiles-$module
     # All dirs in the remote repository.
@@ -470,6 +482,9 @@ for module in $MODULES; do
                         print \"$moduledir/\" \$4}
                 " < $fl > alldirs-$module
+    # Filter the list if requested.
+    filter alldirs-$module
+
     linecount=$(wc -l < allfiles-$module)
     linecount2=$(wc -l < alldirs-$module)
     db2f "Total on server:       %7d files, %4d dirs.\n" $linecount $linecount2
@@ -480,12 +495,18 @@ for module in $MODULES; do
                         print \"$moduledir/\" \$4}
                 " < $fl > newfiles-$module
+    # Filter the list if requested.
+    filter newfiles-$module
+
     # Dirs on the server which changed since the last run
     awk -F '\t' "/\\[Files/ {s=1;next} /^\$/ {s=0;next}
                  { if (s && \$1 >= $LASTTIME &&(\$2 == \"d\"))
                         print \"$moduledir/\" \$4}
                 " < $fl > newdirs-$module
+    # Filter the list if requested.
+    filter newdirs-$module
+
     linecount=$(wc -l < newfiles-$module)
     linecount2=$(wc -l < newdirs-$module)
     db2f "New on server:         %7d files, %4d dirs.\n" $linecount $linecount2

WDYT?

you'd define FILTEREXP as something like:

\,(/i686|/armv7hl|/source|\.iso),d

With https://pagure.io/quick-fedora-mirror/pull-request/28 merged, we can close this, right?

Yep, this is done now.

@tibbs changed the status to Closed
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.