Decide own entries through the chosen directory, judge unchosen relat…

README.md

@@ -373,9 +373,9 @@ Some other points worth knowing:
 - A file which cannot be read, including files in a directory which cannot be read, is reported as could not be read in `-v` output, counted as missing when it is in the audit file and not counted otherwise, and `hashit` exits with `2` rather than `1`.
 - The same file supplied more than once by the same absolute path, such as through `.` and `./sub`, is only counted once. The same file reached through a symbolic link is a different path, so is counted again.
 - Modification times are recorded with `--mtime` but are not compared by an audit.
-- A `sum` file which lists the same file more than once, such as once with `md5sum` and again with `sha1sum`, or as both `./file` and `file`, has every line checked as `md5sum -c` does. Lines are the same file when they name it through the directory the audit is matched through, including through a symbolic link, and a line is only not reported on its own when another line was checked against that file. A file listed with different values of the same hash is reported as modified with a warning. Lines which only name the same file through another directory name different files, so a `sum` file at the root of a tree listing `./README` and `./sub/README` passes when audited from `sub`. When as many files match through another directory as through the one chosen, lines naming the same file through either must agree, as which the `sum` file is relative to is not known. In a `hashdeep` file only the first entry for a path is used, with a warning. In a SQLite database a row for a path already listed with the same values is not counted, with a warning, while one with other values is reported as missing. Entries are only the same path when they name the same file through the directory the audit is matched through, so after a tree is copied elsewhere an absolute entry into the original is audited on its own. When every file matched an absolute entry no directory is chosen, and entries are the same path when they name the same file through the invoked from directory, or for a `sum` file, which has none, through any of them. A SQLite database whose `file_hashes` table does not have a text `filepath` on every row as its primary key compared exactly, as one not created by hashit may not, is refused rather than audited.
+- A `sum` file which lists the same file more than once, such as once with `md5sum` and again with `sha1sum`, or as both `./file` and `file`, has every line checked as `md5sum -c` does. Lines are the same file when they name it through the directory the audit is matched through, including through a symbolic link, and a line is only not reported on its own when another line was checked against that file. A file listed with different values of the same hash is reported as modified with a warning. Lines which only name the same file through another directory name different files, so a `sum` file at the root of a tree listing `./README` and `./sub/README` passes when audited from `sub`. When as many files match through another directory as through the one chosen, lines naming the same file through either must agree, as which the `sum` file is relative to is not known. In a `hashdeep` file only the first entry for a path is used, with a warning. Without an invoked from directory, entries are only the same path when they name it through every directory, as for a `sum` file. In a SQLite database a row for a path already listed with the same values is not counted, with a warning, while one with other values is reported as missing. Entries are only the same path when they name the same file through the directory the audit is matched through, so after a tree is copied elsewhere an absolute entry into the original is audited on its own. When every file matched an absolute entry no directory is chosen, and entries are the same path when they name the same file through the invoked from directory. Without one, as for a `sum` file, nothing decided which directory relative entries are relative to, so the audit is checked through each directory as for a tie. Otherwise a deleted `a` listed beside an identical `s/a` listed absolutely would pass when audited from `s`. So a `sum` file listing files both relatively and absolutely passes in its own directory and fails from another, as `md5sum -c` does. A SQLite database whose `file_hashes` table does not have a text `filepath` on every row as its primary key compared exactly, as one not created by hashit may not, is refused rather than audited.
 - The audit file of an empty directory passes when audited against an empty directory, as does an audit file which is empty or only contains comments, which `md5sum -c` treats as an error. `hashdeep` reports an error for an audit file with no files in it.
-- An audit file written by a shell redirect into the directory being hashed, such as `hashit --format hashdeep . > manifest.txt`, lists itself as an empty file. That entry is ignored when auditing. Without an invoked from directory, as for a `sum` file, an entry is only taken to be the audit file when it names it through the directory the audit is matched through, so another file of the same name, such as `data/data/hashes.md5` listed in `data/hashes.md5` and audited from the parent of `data`, is still audited. `--output manifest.txt` avoids the entry entirely and is recommended, especially for the `sum` format which writes as it goes and so may record part of itself, which then fails when audited from another directory.
+- An audit file written by a shell redirect into the directory being hashed, such as `hashit --format hashdeep . > manifest.txt`, lists itself as an empty file. That entry is ignored when auditing. Unless it names the audit file through the invoked from directory of a `hashdeep` file, an entry is only taken to be the audit file when it names it through the directory the audit is matched through, so another file of the same name is still audited. Examples are `data/data/hashes.md5` listed in `data/hashes.md5` and audited from the parent of `data`, or an empty `t/hashes.md5` listed in `m/hashes.md5` written from `t`. `--output manifest.txt` avoids the entry entirely and is recommended, especially for the `sum` format which writes as it goes and so may record part of itself, which then fails when audited from another directory.
 - Concatenated `hashdeep` files are read, but only the first `## Invoked from:` line is used, so files concatenated from different directories may report moved files. `hashdeep` also fails to audit these.
 
 #### Modification Times
@@ -394,7 +394,7 @@ hashdeep: Audit passed
 
 #### Paths in Audit Files
 
-Relative paths in a `hashdeep` file are resolved against the directory recorded in its `## Invoked from:` line, so an audit file created with `hashit --format hashdeep .` can be audited from anywhere using either a relative or absolute path. When the invoked from directory does not contain the files, such as after they have been copied elsewhere, paths are also resolved against the directory containing the audit file, being where the file really is and then where the path supplied for it is when that is a symbolic link, and then the current directory. An audit is matched through only one of these, the one through which the most files match their records, or the first of those in this order, so a file found only through another, such as a file moved to where the directory containing the audit file places its record, is reported as moved or new. A file counts towards every directory through which a record names it, except a file an absolute record names, and is checked against the record naming it through the chosen directory, so a file which another record also names through a different directory, such as `meta/README` named by `./README` through a manifest kept in `meta`, is still checked against its own record. When as many files match through another directory as through the first of them, which the paths are relative to is not decided by the files on disk, so the audit is checked through each and passes only when it passes through every one, with a warning naming the directory it fails through. Otherwise a tree copied with its manifest kept in `meta`, where `./README` and `./meta/README` both name `meta/README`, could pass after `README` was moved onto `meta/README`. Relative paths in `sum` files have no invoked from directory, so are resolved against the current directory first. Matching is case sensitive on every platform. Paths are always compared as absolute paths, so the path to audit may be supplied as relative or absolute, and `./file` and `file` are the same file. So a manifest stored at the root of a set of files, created from that directory, can be audited from anywhere after the files are copied or shipped elsewhere. Symbolic links in the path to audit are not resolved, so a manifest created through `real/` and audited through a link to it reports every file as moved, as `hashdeep` does. When the current directory was reached through a symbolic link, paths through where it, or a directory containing it, really is are compared through the link, so a manifest of absolute paths through `real/` audited with `.` from the link passes. A path containing `..`, such as `../dir` or `link/../dir`, names the directory the operating system reads, being the parent of the directory a symbolic link points to rather than the directory containing the link, in the paths supplied, the paths recorded in audit files and with `--absolute`. Where removing the `..` names the same directory, such as from a current directory reached through a symbolic link, the path is kept as written, A recorded path through a directory which no longer exists, such as the directory an audit file was created in, has its `..` removed as text, as a missing directory cannot be a symbolic link, while a recorded path with `..` after a link which no longer exists, a file, or a directory which cannot be searched never matches another file. When two files on disk match the same record, such as when the path to audit overlaps a copy of the files, the one matching through the invoked from directory is used and the other is reported as new.
+Relative paths in a `hashdeep` file are resolved against the directory recorded in its `## Invoked from:` line, so an audit file created with `hashit --format hashdeep .` can be audited from anywhere using either a relative or absolute path. When the invoked from directory does not contain the files, such as after they have been copied elsewhere, paths are also resolved against the directory containing the audit file, being where the file really is and then where the path supplied for it is when that is a symbolic link, and then the current directory. An audit is matched through only one of these, the one through which the most files match their records, or the first of those in this order, so a file found only through another, such as a file moved to where the directory containing the audit file places its record, is reported as moved or new. A file counts towards every directory through which a record names it, except a file an absolute record names, and is checked against the record naming it through the chosen directory, so a file which another record also names through a different directory, such as `meta/README` named by `./README` through a manifest kept in `meta`, is still checked against its own record. When as many files match through another directory as through the first of them, which the paths are relative to is not decided by the files on disk, so the audit is checked through each and passes only when it passes through every one. The first directory is reported when it fails, otherwise the one it fails through, with a warning naming it. Otherwise a tree copied with its manifest kept in `meta`, where `./README` and `./meta/README` both name `meta/README`, could pass after `README` was moved onto `meta/README`. Relative paths in `sum` files have no invoked from directory, so are resolved against the current directory first. Matching is case sensitive on every platform. Paths are always compared as absolute paths, so the path to audit may be supplied as relative or absolute, and `./file` and `file` are the same file. So a manifest stored at the root of a set of files, created from that directory, can be audited from anywhere after the files are copied or shipped elsewhere. Symbolic links in the path to audit are not resolved, so a manifest created through `real/` and audited through a link to it reports every file as moved, as `hashdeep` does. When the current directory was reached through a symbolic link, paths through where it, or a directory containing it, really is are compared through the link, so a manifest of absolute paths through `real/` audited with `.` from the link passes. A path containing `..`, such as `../dir` or `link/../dir`, names the directory the operating system reads, being the parent of the directory a symbolic link points to rather than the directory containing the link, in the paths supplied, the paths recorded in audit files and with `--absolute`. Where removing the `..` names the same directory, such as from a current directory reached through a symbolic link, the path is kept as written, A recorded path through a directory which no longer exists, such as the directory an audit file was created in, has its `..` removed as text, as a missing directory cannot be a symbolic link, while a recorded path with `..` after a link which no longer exists, a file, or a directory which cannot be searched never matches another file. When two files on disk match the same record, such as when the path to audit overlaps a copy of the files, the one matching through the invoked from directory is used and the other is reported as new.
 
 If the audit file is inside the directory being audited it is skipped, as it can never contain its own hash. `hashdeep` and `md5sum` do not do this, so when generating a file for them to verify, write it outside the directory being hashed, otherwise they will report the partially written file as changed.
 

main_audit_test.go

@@ -139,6 +139,11 @@ func newAuditScenario(seed uint64) *auditScenario {
 		if r.Intn(8) == 0 {
 			content = "content 0\n" // identical files
 		}
+		if name == s.manifestName() && r.Intn(3) == 0 {
+			// an empty file of the manifest's name, as a manifest written by a shell redirect lists
+			// itself as an empty file
+			content = ""
+		}
 		s.files[p] = content
 	}
 
@@ -799,6 +804,48 @@ var auditRegressions = []struct {
 			files:   map[string]string{"README": "top\n", "meta/README": "meta\n"},
 			changes: []string{"rename meta/README meta/README.old", "rename README meta/README"}},
 	},
+	{
+		// through the manifest's directory the line naming the empty file names the manifest, so it
+		// was dropped as the manifest listing itself before the directory was chosen
+		name:   "unchanged empty file of the manifest's name beside it fails, sum",
+		strict: true,
+		s: &auditScenario{format: "sum", location: "out", relocation: "none", auditFrom: "root",
+			files: map[string]string{"hashes.md5": "", "a": "a\n", "sub/b": "b\n"}},
+	},
+	{
+		name: "deleted empty file of the manifest's name beside it passes, sum",
+		s: &auditScenario{format: "sum", location: "out", relocation: "none", auditFrom: "root",
+			files:   map[string]string{"hashes.md5": "", "a": "a\n", "sub/b": "b\n"},
+			changes: []string{"delete hashes.md5"}},
+	},
+	{
+		// through the manifest's directory the line naming the empty file names the manifest, so it
+		// was dropped as the manifest listing itself before the directory was chosen
+		name:   "unchanged empty file of the manifest's name beside it fails, hashdeep",
+		strict: true,
+		s: &auditScenario{format: "hashdeep", location: "out", relocation: "none", auditFrom: "root",
+			files: map[string]string{"m.hd": "", "a": "a\n", "sub/b": "b\n"}},
+	},
+	{
+		name: "deleted empty file of the manifest's name beside it passes, hashdeep",
+		s: &auditScenario{format: "hashdeep", location: "out", relocation: "none", auditFrom: "root",
+			files:   map[string]string{"m.hd": "", "a": "a\n", "sub/b": "b\n"},
+			changes: []string{"delete m.hd"}},
+	},
+	{
+		// through the manifest's directory the line naming the empty file names the manifest, so it
+		// was dropped as the manifest listing itself before the directory was chosen
+		name:   "unchanged empty file of the manifest's name beside it fails, sqlite",
+		strict: true,
+		s: &auditScenario{format: "sqlite", location: "out", relocation: "none", auditFrom: "root",
+			files: map[string]string{"m.db": "", "a": "a\n", "sub/b": "b\n"}},
+	},
+	{
+		name: "deleted empty file of the manifest's name beside it passes, sqlite",
+		s: &auditScenario{format: "sqlite", location: "out", relocation: "none", auditFrom: "root",
+			files:   map[string]string{"m.db": "", "a": "a\n", "sub/b": "b\n"},
+			changes: []string{"delete m.db"}},
+	},
 }
 
 func TestAuditRegressions(t *testing.T) {