New Upstream Release - golang-github-karrick-godirwalk

Ready changes

Summary

Merged new upstream version: 1.17.0 (was: 1.15.3).

Resulting package

Built on 2022-11-24T13:50 (took 2m57s)

The resulting binary packages can be installed (if you have the apt repository enabled) by running one of:

apt install -t fresh-releases golang-github-karrick-godirwalk-dev

Lintian Result

Diff

diff --git a/README.md b/README.md
index 72c51a5..ccfb2cb 100644
--- a/README.md
+++ b/README.md
@@ -5,13 +5,17 @@ system.
 
 [![GoDoc](https://godoc.org/github.com/karrick/godirwalk?status.svg)](https://godoc.org/github.com/karrick/godirwalk) [![Build Status](https://dev.azure.com/microsoft0235/microsoft/_apis/build/status/karrick.godirwalk?branchName=master)](https://dev.azure.com/microsoft0235/microsoft/_build/latest?definitionId=1&branchName=master)
 
-In short, why do I use this library?
+In short, why did I create this library?
 
 1. It's faster than `filepath.Walk`.
 1. It's more correct on Windows than `filepath.Walk`.
 1. It's more easy to use than `filepath.Walk`.
 1. It's more flexible than `filepath.Walk`.
 
+Depending on your specific circumstances, [you might no longer need a
+library for file walking in
+Go](https://engineering.kablamo.com.au/posts/2021/quick-comparison-between-go-file-walk-implementations).
+
 ## Usage Example
 
 Additional examples are provided in the `examples/` subdirectory.
@@ -26,6 +30,12 @@ provided callback function.
     dirname := "some/directory/root"
     err := godirwalk.Walk(dirname, &godirwalk.Options{
         Callback: func(osPathname string, de *godirwalk.Dirent) error {
+            // Following string operation is not most performant way
+            // of doing this, but common enough to warrant a simple
+            // example here:
+            if strings.Contains(osPathname, ".git") {
+                return godirwalk.SkipThis
+            }
             fmt.Printf("%s %s\n", de.ModeType(), osPathname)
             return nil
         },
@@ -47,8 +57,8 @@ Here's why I use `godirwalk` in preference to `filepath.Walk`,
 
 When compared against `filepath.Walk` in benchmarks, it has been
 observed to run between five and ten times the speed on darwin, at
-speeds comparable to the that of the unix `find` utility; about twice
-the speed on linux; and about four times the speed on Windows.
+speeds comparable to the that of the unix `find` utility; and about
+twice the speed on linux; and about four times the speed on Windows.
 
 How does it obtain this performance boost? It does less work to give
 you nearly the same output. This library calls the same `syscall`
@@ -60,11 +70,11 @@ file system entry data from the operating system.
 
 While traversing a file system directory tree, `filepath.Walk` obtains
 the list of immediate descendants of a directory, and throws away the
-file system node type information provided by the operating system
-that comes with the node's name. Then, immediately prior to invoking
-the callback function, `filepath.Walk` invokes `os.Stat` for each
-node, and passes the returned `os.FileInfo` information to the
-callback.
+node type information for the file system entry that is provided by
+the operating system that comes with the node's name. Then,
+immediately prior to invoking the callback function, `filepath.Walk`
+invokes `os.Stat` for each node, and passes the returned `os.FileInfo`
+information to the callback.
 
 While the `os.FileInfo` information provided by `os.Stat` is extremely
 helpful--and even includes the `os.FileMode` data--providing it
@@ -141,13 +151,20 @@ The takeaway is that behavior is different based on which platform
 until it is fixed in the standard library, it presents a compatibility
 problem.
 
-This library correctly identifies symbolic links that point to
-directories and will only follow them when `FollowSymbolicLinks` is
-set to true. Behavior on Windows and other operating systems is
-identical.
+This library fixes the above problem such that it will never follow
+logical file sytem loops on either unix or Windows. Furthermore, it
+will only follow symbolic links when `FollowSymbolicLinks` is set to
+true. Behavior on Windows and other operating systems is identical.
 
 ### It's more easy to use than `filepath.Walk`
 
+While this library strives to mimic the behavior of the incredibly
+well-written `filepath.Walk` standard library, there are places where
+it deviates a bit in order to provide a more easy or intuitive caller
+interface.
+
+#### Callback interface does not send you an error to check
+
 Since this library does not invoke `os.Stat` on every file system node
 it encounters, there is no possible error event for the callback
 function to filter on. The third argument in the `filepath.WalkFunc`
@@ -155,23 +172,105 @@ function signature to pass the error from `os.Stat` to the callback
 function is no longer necessary, and thus eliminated from signature of
 the callback function from this library.
 
-Also, `filepath.Walk` invokes the callback function with a solidus
-delimited pathname regardless of the os-specific path separator. This
-library invokes the callback function with the os-specific pathname
-separator, obviating a call to `filepath.Clean` in the callback
-function for each node prior to actually using the provided pathname.
+Furthermore, this slight interface difference between
+`filepath.WalkFunc` and this library's `WalkFunc` eliminates the
+boilerplate code that callback handlers must write when they use
+`filepath.Walk`. Rather than every callback function needing to check
+the error value passed into it and branch accordingly, users of this
+library do not even have an error value to check immediately upon
+entry into the callback function. This is an improvement both in
+runtime performance and code clarity.
+
+#### Callback function is invoked with OS specific file system path separator
+
+On every OS platform `filepath.Walk` invokes the callback function
+with a solidus (`/`) delimited pathname. By contrast this library
+invokes the callback with the os-specific pathname separator,
+obviating a call to `filepath.Clean` in the callback function for each
+node prior to actually using the provided pathname.
 
 In other words, even on Windows, `filepath.Walk` will invoke the
 callback with `some/path/to/foo.txt`, requiring well written clients
 to perform pathname normalization for every file prior to working with
-the specified file. In truth, many clients developed on unix and not
-tested on Windows neglect this subtlety, and will result in software
-bugs when running on Windows. This library would invoke the callback
-function with `some\path\to\foo.txt` for the same file when running on
-Windows, eliminating the need to normalize the pathname by the client,
-and lessen the likelyhood that a client will work on unix but not on
+the specified file. This is a hidden boilerplate requirement to create
+truly os agnostic callback functions. In truth, many clients developed
+on unix and not tested on Windows neglect this subtlety, and will
+result in software bugs when someone tries to run that software on
 Windows.
 
+This library invokes the callback function with `some\path\to\foo.txt`
+for the same file when running on Windows, eliminating the need to
+normalize the pathname by the client, and lessen the likelyhood that a
+client will work on unix but not on Windows.
+
+This enhancement eliminates necessity for some more boilerplate code
+in callback functions while improving the runtime performance of this
+library.
+
+#### `godirwalk.SkipThis` is more intuitive to use than `filepath.SkipDir`
+
+One arguably confusing aspect of the `filepath.WalkFunc` interface
+that this library must emulate is how a caller tells the `Walk`
+function to skip file system entries. With both `filepath.Walk` and
+this library's `Walk`, when a callback function wants to skip a
+directory and not descend into its children, it returns
+`filepath.SkipDir`. If the callback function returns
+`filepath.SkipDir` for a non-directory, `filepath.Walk` and this
+library will stop processing any more entries in the current
+directory. This is not necessarily what most developers want or
+expect. If you want to simply skip a particular non-directory entry
+but continue processing entries in the directory, the callback
+function must return nil.
+
+The implications of this interface design is when you want to walk a
+file system hierarchy and skip an entry, you have to return a
+different value based on what type of file system entry that node
+is. To skip an entry, if the entry is a directory, you must return
+`filepath.SkipDir`, and if entry is not a directory, you must return
+`nil`. This is an unfortunate hurdle I have observed many developers
+struggling with, simply because it is not an intuitive interface.
+
+Here is an example callback function that adheres to
+`filepath.WalkFunc` interface to have it skip any file system entry
+whose full pathname includes a particular substring, `optSkip`. Note
+that this library still supports identical behavior of `filepath.Walk`
+when the callback function returns `filepath.SkipDir`.
+
+```Go
+    func callback1(osPathname string, de *godirwalk.Dirent) error {
+        if optSkip != "" && strings.Contains(osPathname, optSkip) {
+            if b, err := de.IsDirOrSymlinkToDir(); b == true && err == nil {
+                return filepath.SkipDir
+            }
+            return nil
+        }
+        // Process file like normal...
+        return nil
+    }
+```
+
+This library attempts to eliminate some of that logic boilerplate
+required in callback functions by providing a new token error value,
+`SkipThis`, which a callback function may return to skip the current
+file system entry regardless of what type of entry it is. If the
+current entry is a directory, its children will not be enumerated,
+exactly as if the callback had returned `filepath.SkipDir`. If the
+current entry is a non-directory, the next file system entry in the
+current directory will be enumerated, exactly as if the callback
+returned `nil`. The following example callback function has identical
+behavior as the previous, but has less boilerplate, and admittedly
+logic that I find more simple to follow.
+
+```Go
+    func callback2(osPathname string, de *godirwalk.Dirent) error {
+        if optSkip != "" && strings.Contains(osPathname, optSkip) {
+            return godirwalk.SkipThis
+        }
+        // Process file like normal...
+        return nil
+    }
+```
+
 ### It's more flexible than `filepath.Walk`
 
 #### Configurable Handling of Symbolic Links
@@ -182,7 +281,7 @@ does. However, it does invoke the callback function with each node it
 finds, including symbolic links. If a particular use case exists to
 follow symbolic links when traversing a directory tree, this library
 can be invoked in manner to do so, by setting the
-`FollowSymbolicLinks` parameter to true.
+`FollowSymbolicLinks` config parameter to `true`.
 
 #### Configurable Sorting of Directory Children
 
@@ -191,11 +290,12 @@ descendants of a directory prior to visiting each node, just like
 `filepath.Walk` does. This is usually the desired behavior. However,
 this does come at slight performance and memory penalties required to
 sort the names when a directory node has many entries. Additionally if
-caller specifies `Unsorted` enumeration, reading directories is lazily
-performed as the caller consumes entries. If a particular use case
-exists that does not require sorting the directory's immediate
-descendants prior to visiting its nodes, this library will skip the
-sorting step when the `Unsorted` parameter is set to true.
+caller specifies `Unsorted` enumeration in the configuration
+parameter, reading directories is lazily performed as the caller
+consumes entries. If a particular use case exists that does not
+require sorting the directory's immediate descendants prior to
+visiting its nodes, this library will skip the sorting step when the
+`Unsorted` parameter is set to `true`.
 
 Here's an interesting read of the potential hazzards of traversing a
 file system hierarchy in a non-deterministic order. If you know the
@@ -208,10 +308,11 @@ setting this option.
 #### Configurable Post Children Callback
 
 This library provides upstream code with the ability to specify a
-callback to be invoked for each directory after its children are
-processed. This has been used to recursively delete empty directories
-after traversing the file system in a more efficient manner. See the
-`examples/clean-empties` directory for an example of this usage.
+callback function to be invoked for each directory after its children
+are processed. This has been used to recursively delete empty
+directories after traversing the file system in a more efficient
+manner. See the `examples/clean-empties` directory for an example of
+this usage.
 
 #### Configurable Error Callback
 
diff --git a/bench.sh b/bench.sh
new file mode 100755
index 0000000..b2ba374
--- /dev/null
+++ b/bench.sh
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# for version in v1.9.1 v1.10.0 v1.10.3 v1.10.12 v1.11.2 v1.11.3 v1.12.0 v1.13.1 v1.14.0 v1.14.1 ; do
+for version in v1.10.12 v1.14.1 v1.15.2 ; do
+    echo "### $version" > $version.txt
+    git checkout -- go.mod && git checkout $version && go test -run=NONE -bench=Benchmark2 >> $version.txt || exit 1
+done
diff --git a/benchmark_test.go b/benchmark_test.go
new file mode 100644
index 0000000..f052111
--- /dev/null
+++ b/benchmark_test.go
@@ -0,0 +1,85 @@
+package godirwalk
+
+import (
+	"path/filepath"
+	"testing"
+)
+
+const benchRoot = "/mnt/ram_disk/src"
+
+var scratch []byte
+var largeDirectory string
+
+func init() {
+	scratch = make([]byte, MinimumScratchBufferSize)
+	largeDirectory = filepath.Join(benchRoot, "linkedin/dashboards")
+}
+
+func Benchmark2ReadDirentsGodirwalk(b *testing.B) {
+	var count int
+
+	for i := 0; i < b.N; i++ {
+		actual, err := ReadDirents(largeDirectory, scratch)
+		if err != nil {
+			b.Fatal(err)
+		}
+		count += len(actual)
+	}
+
+	_ = count
+}
+
+func Benchmark2ReadDirnamesGodirwalk(b *testing.B) {
+	var count int
+
+	for i := 0; i < b.N; i++ {
+		actual, err := ReadDirnames(largeDirectory, scratch)
+		if err != nil {
+			b.Fatal(err)
+		}
+		count += len(actual)
+	}
+
+	_ = count
+}
+
+func Benchmark2GodirwalkSorted(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		var length int
+		err := Walk(benchRoot, &Options{
+			Callback: func(name string, _ *Dirent) error {
+				if name == "skip" {
+					return filepath.SkipDir
+				}
+				length += len(name)
+				return nil
+			},
+			ScratchBuffer: scratch,
+		})
+		if err != nil {
+			b.Errorf("GOT: %v; WANT: nil", err)
+		}
+		_ = length
+	}
+}
+
+func Benchmark2GodirwalkUnsorted(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		var length int
+		err := Walk(benchRoot, &Options{
+			Callback: func(name string, _ *Dirent) error {
+				if name == "skip" {
+					return filepath.SkipDir
+				}
+				length += len(name)
+				return nil
+			},
+			ScratchBuffer: scratch,
+			Unsorted:      true,
+		})
+		if err != nil {
+			b.Errorf("GOT: %v; WANT: nil", err)
+		}
+		_ = length
+	}
+}
diff --git a/debian/changelog b/debian/changelog
index 8ac495d..f987a36 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+golang-github-karrick-godirwalk (1.17.0-1) UNRELEASED; urgency=low
+
+  * New upstream release.
+
+ -- Debian Janitor <janitor@jelmer.uk>  Thu, 24 Nov 2022 13:48:06 -0000
+
 golang-github-karrick-godirwalk (1.15.3-3) unstable; urgency=medium
 
   [ Debian Janitor ]
diff --git a/examples/find-fast/main.go b/examples/find-fast/main.go
index 9211f47..c7d1e21 100644
--- a/examples/find-fast/main.go
+++ b/examples/find-fast/main.go
@@ -10,6 +10,7 @@ import (
 	"os"
 	"path/filepath"
 	"regexp"
+	"strings"
 
 	"github.com/karrick/godirwalk"
 	"github.com/karrick/golf"
@@ -19,8 +20,9 @@ import (
 var NoColor = os.Getenv("TERM") == "dumb" || !(isatty.IsTerminal(os.Stdout.Fd()) || isatty.IsCygwinTerminal(os.Stdout.Fd()))
 
 func main() {
-	optRegex := golf.String("regex", "", "Do not print unless full path matches regex.")
 	optQuiet := golf.Bool("quiet", false, "Do not print intermediate errors to stderr.")
+	optRegex := golf.String("regex", "", "Do not print unless full path matches regex.")
+	optSkip := golf.String("skip", "", "Skip and do not descend into entries with this substring in the pathname")
 	golf.Parse()
 
 	programName, err := os.Executable()
@@ -53,13 +55,25 @@ func main() {
 	switch {
 	case nameRE == nil:
 		// When no name pattern provided, print everything.
-		options.Callback = func(osPathname string, _ *godirwalk.Dirent) error {
+		options.Callback = func(osPathname string, de *godirwalk.Dirent) error {
+			if *optSkip != "" && strings.Contains(osPathname, *optSkip) {
+				if !*optQuiet {
+					fmt.Fprintf(os.Stderr, "%s: %s (skipping)\n", programName, osPathname)
+				}
+				return godirwalk.SkipThis
+			}
 			_, err := fmt.Println(osPathname)
 			return err
 		}
 	case NoColor:
 		// Name pattern was provided, but color not permitted.
 		options.Callback = func(osPathname string, _ *godirwalk.Dirent) error {
+			if *optSkip != "" && strings.Contains(osPathname, *optSkip) {
+				if !*optQuiet {
+					fmt.Fprintf(os.Stderr, "%s: %s (skipping)\n", programName, osPathname)
+				}
+				return godirwalk.SkipThis
+			}
 			var err error
 			if nameRE.FindString(osPathname) != "" {
 				_, err = fmt.Println(osPathname)
@@ -71,6 +85,12 @@ func main() {
 		buf = append(buf, "\033[22m"...) // very first print should set normal intensity
 
 		options.Callback = func(osPathname string, _ *godirwalk.Dirent) error {
+			if *optSkip != "" && strings.Contains(osPathname, *optSkip) {
+				if !*optQuiet {
+					fmt.Fprintf(os.Stderr, "%s: %s (skipping)\n", programName, osPathname)
+				}
+				return godirwalk.SkipThis
+			}
 			matches := nameRE.FindAllStringSubmatchIndex(osPathname, -1)
 			if len(matches) == 0 {
 				return nil // entry does not match pattern
diff --git a/readdir_unix.go b/readdir_unix.go
index 8c9404c..a993038 100644
--- a/readdir_unix.go
+++ b/readdir_unix.go
@@ -31,11 +31,15 @@ func readDirents(osDirname string, scratchBuffer []byte) ([]*Dirent, error) {
 		scratchBuffer = newScratchBuffer()
 	}
 
+	var sde syscall.Dirent
 	for {
 		if len(workBuffer) == 0 {
 			n, err := syscall.ReadDirent(fd, scratchBuffer)
 			// n, err := unix.ReadDirent(fd, scratchBuffer)
 			if err != nil {
+				if err == syscall.EINTR /* || err == unix.EINTR */ {
+					continue
+				}
 				_ = dh.Close()
 				return nil, err
 			}
@@ -48,14 +52,14 @@ func readDirents(osDirname string, scratchBuffer []byte) ([]*Dirent, error) {
 			workBuffer = scratchBuffer[:n] // trim work buffer to number of bytes read
 		}
 
-		sde := (*syscall.Dirent)(unsafe.Pointer(&workBuffer[0])) // point entry to first syscall.Dirent in buffer
-		workBuffer = workBuffer[reclen(sde):]                    // advance buffer for next iteration through loop
+		copy((*[unsafe.Sizeof(syscall.Dirent{})]byte)(unsafe.Pointer(&sde))[:], workBuffer)
+		workBuffer = workBuffer[reclen(&sde):] // advance buffer for next iteration through loop
 
-		if inoFromDirent(sde) == 0 {
+		if inoFromDirent(&sde) == 0 {
 			continue // inode set to 0 indicates an entry that was marked as deleted
 		}
 
-		nameSlice := nameFromDirent(sde)
+		nameSlice := nameFromDirent(&sde)
 		nameLength := len(nameSlice)
 
 		if nameLength == 0 || (nameSlice[0] == '.' && (nameLength == 1 || (nameLength == 2 && nameSlice[1] == '.'))) {
@@ -63,7 +67,7 @@ func readDirents(osDirname string, scratchBuffer []byte) ([]*Dirent, error) {
 		}
 
 		childName := string(nameSlice)
-		mt, err := modeTypeFromDirent(sde, osDirname, childName)
+		mt, err := modeTypeFromDirent(&sde, osDirname, childName)
 		if err != nil {
 			_ = dh.Close()
 			return nil, err
@@ -92,6 +96,9 @@ func readDirnames(osDirname string, scratchBuffer []byte) ([]string, error) {
 			n, err := syscall.ReadDirent(fd, scratchBuffer)
 			// n, err := unix.ReadDirent(fd, scratchBuffer)
 			if err != nil {
+				if err == syscall.EINTR /* || err == unix.EINTR */ {
+					continue
+				}
 				_ = dh.Close()
 				return nil, err
 			}
@@ -104,9 +111,9 @@ func readDirnames(osDirname string, scratchBuffer []byte) ([]string, error) {
 			workBuffer = scratchBuffer[:n] // trim work buffer to number of bytes read
 		}
 
-		// Handle first entry in the work buffer.
 		sde = (*syscall.Dirent)(unsafe.Pointer(&workBuffer[0])) // point entry to first syscall.Dirent in buffer
-		workBuffer = workBuffer[reclen(sde):]                   // advance buffer for next iteration through loop
+		// Handle first entry in the work buffer.
+		workBuffer = workBuffer[reclen(sde):] // advance buffer for next iteration through loop
 
 		if inoFromDirent(sde) == 0 {
 			continue // inode set to 0 indicates an entry that was marked as deleted
diff --git a/scandir_unix.go b/scandir_unix.go
index 46a06a6..654039b 100644
--- a/scandir_unix.go
+++ b/scandir_unix.go
@@ -1,3 +1,4 @@
+//go:build !windows
 // +build !windows
 
 package godirwalk
@@ -18,12 +19,14 @@ type Scanner struct {
 	statErr       error    // statErr is any error return while attempting to stat an entry
 	dh            *os.File // used to close directory after done reading
 	de            *Dirent  // most recently decoded directory entry
-	sde           *syscall.Dirent
+	sde           syscall.Dirent
 	fd            int // file descriptor used to read entries from directory
 }
 
-// NewScanner returns a new directory Scanner that lazily enumerates the
-// contents of a single directory.
+// NewScanner returns a new directory Scanner that lazily enumerates
+// the contents of a single directory. To prevent resource leaks,
+// caller must invoke either the Scanner's Close or Err method after
+// it has completed scanning a directory.
 //
 //     scanner, err := godirwalk.NewScanner(dirname)
 //     if err != nil {
@@ -52,10 +55,12 @@ func NewScanner(osDirname string) (*Scanner, error) {
 	return NewScannerWithScratchBuffer(osDirname, nil)
 }
 
-// NewScannerWithScratchBuffer returns a new directory Scanner that lazily
-// enumerates the contents of a single directory. On platforms other than
-// Windows it uses the provided scratch buffer to read from the file system. On
-// Windows the scratch buffer is ignored.
+// NewScannerWithScratchBuffer returns a new directory Scanner that
+// lazily enumerates the contents of a single directory. On platforms
+// other than Windows it uses the provided scratch buffer to read from
+// the file system. On Windows the scratch buffer is ignored. To
+// prevent resource leaks, caller must invoke either the Scanner's
+// Close or Err method after it has completed scanning a directory.
 func NewScannerWithScratchBuffer(osDirname string, scratchBuffer []byte) (*Scanner, error) {
 	dh, err := os.Open(osDirname)
 	if err != nil {
@@ -73,11 +78,18 @@ func NewScannerWithScratchBuffer(osDirname string, scratchBuffer []byte) (*Scann
 	return scanner, nil
 }
 
+// Close releases resources associated with scanning a directory. Call
+// either this or the Err method when the directory no longer needs to
+// be scanned.
+func (s *Scanner) Close() error {
+	return s.Err()
+}
+
 // Dirent returns the current directory entry while scanning a directory.
 func (s *Scanner) Dirent() (*Dirent, error) {
 	if s.de == nil {
 		s.de = &Dirent{name: s.childName, path: s.osDirname}
-		s.de.modeType, s.statErr = modeTypeFromDirent(s.sde, s.osDirname, s.childName)
+		s.de.modeType, s.statErr = modeTypeFromDirent(&s.sde, s.osDirname, s.childName)
 	}
 	return s.de, s.statErr
 }
@@ -90,19 +102,23 @@ func (s *Scanner) done(err error) {
 		return
 	}
 
-	if cerr := s.dh.Close(); err == nil {
-		s.err = cerr
+	s.err = err
+
+	if err = s.dh.Close(); s.err == nil {
+		s.err = err
 	}
 
 	s.osDirname, s.childName = "", ""
 	s.scratchBuffer, s.workBuffer = nil, nil
-	s.dh, s.de, s.sde, s.statErr = nil, nil, nil, nil
+	s.dh, s.de, s.statErr = nil, nil, nil
+	s.sde = syscall.Dirent{}
 	s.fd = 0
 }
 
-// Err returns any error associated with scanning a directory. It is normal to
-// call Err after Scan returns false, even though they both ensure Scanner
-// resources are released. Do not call until done scanning a directory.
+// Err returns any error associated with scanning a directory. It is
+// normal to call Err after Scan returns false, even though they both
+// ensure Scanner resources are released. Call either this or the
+// Close method when the directory no longer needs to be scanned.
 func (s *Scanner) Err() error {
 	s.done(nil)
 	return s.err
@@ -131,7 +147,10 @@ func (s *Scanner) Scan() bool {
 			n, err := syscall.ReadDirent(s.fd, s.scratchBuffer)
 			// n, err := unix.ReadDirent(s.fd, s.scratchBuffer)
 			if err != nil {
-				s.done(err)
+				if err == syscall.EINTR /* || err == unix.EINTR */ {
+					continue
+				}
+				s.done(err) // any other error forces a stop
 				return false
 			}
 			if n <= 0 { // end of directory: normal exit
@@ -141,14 +160,15 @@ func (s *Scanner) Scan() bool {
 			s.workBuffer = s.scratchBuffer[:n] // trim work buffer to number of bytes read
 		}
 
-		s.sde = (*syscall.Dirent)(unsafe.Pointer(&s.workBuffer[0])) // point entry to first syscall.Dirent in buffer
-		s.workBuffer = s.workBuffer[reclen(s.sde):]                 // advance buffer for next iteration through loop
+		// point entry to first syscall.Dirent in buffer
+		copy((*[unsafe.Sizeof(syscall.Dirent{})]byte)(unsafe.Pointer(&s.sde))[:], s.workBuffer)
+		s.workBuffer = s.workBuffer[reclen(&s.sde):] // advance buffer for next iteration through loop
 
-		if inoFromDirent(s.sde) == 0 {
+		if inoFromDirent(&s.sde) == 0 {
 			continue // inode set to 0 indicates an entry that was marked as deleted
 		}
 
-		nameSlice := nameFromDirent(s.sde)
+		nameSlice := nameFromDirent(&s.sde)
 		nameLength := len(nameSlice)
 
 		if nameLength == 0 || (nameSlice[0] == '.' && (nameLength == 1 || (nameLength == 2 && nameSlice[1] == '.'))) {
diff --git a/scandir_windows.go b/scandir_windows.go
index a211061..338e323 100644
--- a/scandir_windows.go
+++ b/scandir_windows.go
@@ -1,3 +1,4 @@
+//go:build windows
 // +build windows
 
 package godirwalk
@@ -17,8 +18,10 @@ type Scanner struct {
 	childMode os.FileMode
 }
 
-// NewScanner returns a new directory Scanner that lazily enumerates the
-// contents of a single directory.
+// NewScanner returns a new directory Scanner that lazily enumerates
+// the contents of a single directory. To prevent resource leaks,
+// caller must invoke either the Scanner's Close or Err method after
+// it has completed scanning a directory.
 //
 //     scanner, err := godirwalk.NewScanner(dirname)
 //     if err != nil {
@@ -55,14 +58,24 @@ func NewScanner(osDirname string) (*Scanner, error) {
 	return scanner, nil
 }
 
-// NewScannerWithScratchBuffer returns a new directory Scanner that lazily
-// enumerates the contents of a single directory. On platforms other than
-// Windows it uses the provided scratch buffer to read from the file system. On
-// Windows the scratch buffer parameter is ignored.
+// NewScannerWithScratchBuffer returns a new directory Scanner that
+// lazily enumerates the contents of a single directory. On platforms
+// other than Windows it uses the provided scratch buffer to read from
+// the file system. On Windows the scratch buffer parameter is
+// ignored. To prevent resource leaks, caller must invoke either the
+// Scanner's Close or Err method after it has completed scanning a
+// directory.
 func NewScannerWithScratchBuffer(osDirname string, scratchBuffer []byte) (*Scanner, error) {
 	return NewScanner(osDirname)
 }
 
+// Close releases resources associated with scanning a directory. Call
+// either this or the Err method when the directory no longer needs to
+// be scanned.
+func (s *Scanner) Close() error {
+	return s.Err()
+}
+
 // Dirent returns the current directory entry while scanning a directory.
 func (s *Scanner) Dirent() (*Dirent, error) {
 	if s.de == nil {
@@ -83,17 +96,20 @@ func (s *Scanner) done(err error) {
 		return
 	}
 
-	if cerr := s.dh.Close(); err == nil {
-		s.err = cerr
+	s.err = err
+
+	if err = s.dh.Close(); s.err == nil {
+		s.err = err
 	}
 
 	s.childName, s.osDirname = "", ""
 	s.de, s.dh = nil, nil
 }
 
-// Err returns any error associated with scanning a directory. It is normal to
-// call Err after Scan returns false, even though they both ensure Scanner
-// resources are released. Do not call until done scanning a directory.
+// Err returns any error associated with scanning a directory. It is
+// normal to call Err after Scan returns false, even though they both
+// ensure Scanner resources are released. Call either this or the
+// Close method when the directory no longer needs to be scanned.
 func (s *Scanner) Err() error {
 	s.done(nil)
 	return s.err
diff --git a/walk.go b/walk.go
index b15a190..9d0235d 100644
--- a/walk.go
+++ b/walk.go
@@ -96,6 +96,11 @@ const (
 	SkipNode
 )
 
+// SkipThis is used as a return value from WalkFuncs to indicate that the file
+// system entry named in the call is to be skipped. It is not returned as an
+// error by any function.
+var SkipThis = errors.New("skip this directory entry")
+
 // WalkFunc is the type of the function called for each file system node visited
 // by Walk. The pathname argument will contain the argument to Walk as a prefix;
 // that is, if Walk is called with "dir", which is a directory containing the
@@ -119,6 +124,55 @@ const (
 // Walk skips the remaining files in the containing directory. Note that any
 // supplied ErrorCallback function is not invoked with filepath.SkipDir when the
 // Callback or PostChildrenCallback functions return that special value.
+//
+// One arguably confusing aspect of the filepath.WalkFunc API that this library
+// must emulate is how a caller tells Walk to skip file system entries or
+// directories. With both filepath.Walk and this Walk, when a callback function
+// wants to skip a directory and not descend into its children, it returns
+// filepath.SkipDir. If the callback function returns filepath.SkipDir for a
+// non-directory, filepath.Walk and this library will stop processing any more
+// entries in the current directory, which is what many people do not want. If
+// you want to simply skip a particular non-directory entry but continue
+// processing entries in the directory, a callback function must return nil. The
+// implications of this API is when you want to walk a file system hierarchy and
+// skip an entry, when the entry is a directory, you must return one value,
+// namely filepath.SkipDir, but when the entry is a non-directory, you must
+// return a different value, namely nil. In other words, to get identical
+// behavior for two file system entry types you need to send different token
+// values.
+//
+// Here is an example callback function that adheres to filepath.Walk API to
+// have it skip any file system entry whose full pathname includes a particular
+// substring, optSkip:
+//
+//     func callback1(osPathname string, de *godirwalk.Dirent) error {
+//         if optSkip != "" && strings.Contains(osPathname, optSkip) {
+//             if b, err := de.IsDirOrSymlinkToDir(); b == true && err == nil {
+//                 return filepath.SkipDir
+//             }
+//             return nil
+//         }
+//         // Process file like normal...
+//         return nil
+//     }
+//
+// This library attempts to eliminate some of that logic boilerplate by
+// providing a new token error value, SkipThis, which a callback function may
+// return to skip the current file system entry regardless of what type of entry
+// it is. If the current entry is a directory, its children will not be
+// enumerated, exactly as if the callback returned filepath.SkipDir. If the
+// current entry is a non-directory, the next file system entry in the current
+// directory will be enumerated, exactly as if the callback returned nil. The
+// following example callback function has identical behavior as the previous,
+// but has less boilerplate, and admittedly more simple logic.
+//
+//     func callback2(osPathname string, de *godirwalk.Dirent) error {
+//         if optSkip != "" && strings.Contains(osPathname, optSkip) {
+//             return godirwalk.SkipThis
+//         }
+//         // Process file like normal...
+//         return nil
+//     }
 type WalkFunc func(osPathname string, directoryEntry *Dirent) error
 
 // Walk walks the file tree rooted at the specified directory, calling the
@@ -201,10 +255,15 @@ func Walk(pathname string, options *Options) error {
 		options.ErrorCallback = defaultErrorCallback
 	}
 
-	if err = walk(pathname, dirent, options); err != filepath.SkipDir {
+	err = walk(pathname, dirent, options)
+	switch err {
+	case nil, SkipThis, filepath.SkipDir:
+		// silence SkipThis and filepath.SkipDir for top level
+		debug("no error of significance: %v\n", err)
+		return nil
+	default:
 		return err
 	}
-	return nil // silence SkipDir for top level
 }
 
 // defaultErrorCallback always returns Halt because if the upstream code did not
@@ -217,7 +276,7 @@ func defaultErrorCallback(_ string, _ error) ErrorAction { return Halt }
 func walk(osPathname string, dirent *Dirent, options *Options) error {
 	err := options.Callback(osPathname, dirent)
 	if err != nil {
-		if err == filepath.SkipDir {
+		if err == SkipThis || err == filepath.SkipDir {
 			return err
 		}
 		if action := options.ErrorCallback(osPathname, err); action == SkipNode {
@@ -278,7 +337,7 @@ func walk(osPathname string, dirent *Dirent, options *Options) error {
 		}
 		err = walk(osChildname, deChild, options)
 		debug("osChildname: %q; error: %v\n", osChildname, err)
-		if err == nil {
+		if err == nil || err == SkipThis {
 			continue
 		}
 		if err != filepath.SkipDir {
diff --git a/walk_test.go b/walk_test.go
index 9a91814..4ba7398 100644
--- a/walk_test.go
+++ b/walk_test.go
@@ -136,6 +136,44 @@ func TestWalkSkipDir(t *testing.T) {
 	})
 }
 
+func TestWalkSkipThis(t *testing.T) {
+	t.Run("SkipThis", func(t *testing.T) {
+		var actual []string
+		err := Walk(filepath.Join(scaffolingRoot, "d0"), &Options{
+			Callback: func(osPathname string, dirent *Dirent) error {
+				switch name := dirent.Name(); name {
+				case "skips", "skip", "nothing":
+					return SkipThis
+				}
+				actual = append(actual, filepath.FromSlash(osPathname))
+				return nil
+			},
+			FollowSymbolicLinks: true,
+		})
+
+		ensureError(t, err)
+
+		expected := []string{
+			filepath.Join(scaffolingRoot, "d0"),
+			filepath.Join(scaffolingRoot, "d0/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"),
+			filepath.Join(scaffolingRoot, "d0/f1"),
+			filepath.Join(scaffolingRoot, "d0/d1"),
+			filepath.Join(scaffolingRoot, "d0/d1/f2"),
+			filepath.Join(scaffolingRoot, "d0/symlinks"),
+			filepath.Join(scaffolingRoot, "d0/symlinks/d4"),
+			filepath.Join(scaffolingRoot, "d0/symlinks/d4/toSD1"),
+			filepath.Join(scaffolingRoot, "d0/symlinks/d4/toSD1/f2"),
+			filepath.Join(scaffolingRoot, "d0/symlinks/d4/toSF1"),
+			filepath.Join(scaffolingRoot, "d0/symlinks/toAbs"),
+			filepath.Join(scaffolingRoot, "d0/symlinks/toD1"),
+			filepath.Join(scaffolingRoot, "d0/symlinks/toD1/f2"),
+			filepath.Join(scaffolingRoot, "d0/symlinks/toF1"),
+		}
+
+		ensureStringSlicesMatch(t, actual, expected)
+	})
+}
+
 func TestWalkFollowSymbolicLinks(t *testing.T) {
 	var actual []string
 	var errorCallbackVisited bool

Debdiff

[The following lists of changes regard files as different if they have different names, permissions or owners.]

Files in second set of .debs but not in first

-rw-r--r--  root/root   /usr/share/gocode/src/github.com/karrick/godirwalk/benchmark_test.go

No differences were encountered in the control files

More details

Full run details