SSH.NET SFTP Get Directory List and Files Recursively

Question

I use the Renci.SshNet library to recursively retrieve a list of files and directories using SFTP. I can connect to the SFTP site, but I'm not sure how to get the list of directories and files recursively in C #. I did not find any useful examples.

Has anyone tried this? If so, can you post sample code on how to get these files and folders recursively.

Thanks,
Pravdin

Answer 1

There are some quirks in this library that make this recursive list complex, because the interaction between ChangeDirectory and ListDirectory does not work as you might expect.

The following is not a list of files in the / home directory, but a list of files in the / (root) directory:

 sftp.ChangeDirectory("home"); sftp.ListDirectory("").Select (s => s.FullName); 

The following fails and returns a SftpPathNotFoundException:

 sftp.ChangeDirectory("home"); sftp.ListDirectory("home").Select (s => s.FullName); 

The following is the correct way to list files in the / home directory

 sftp.ChangeDirectory("/"); sftp.ListDirectory("home").Select (s => s.FullName); 

This is pretty crazy if you ask me. Setting the default directory using the ChangeDirectory method ChangeDirectory not affect the ListDirectory method unless you specify a folder in the parameter of this method. It seems that an error should be written for this.

So, when you write your recursive function, you will need to set the default directory once, and then change the directory in the ListDirectory call when you ListDirectory over the folders. The listing lists the SftpFiles. Then they can be checked individually for IsDirectory == true . Just keep in mind that listing also returns entries . and .. (which are directories). You will want to skip them if you want to avoid an infinite loop. :-)

EDIT 2/23/2018

I looked through some of my old answers and would like to apologize for the answer above and provide the following working code. Note that this example does not require ChangeDirectory , since it uses Fullname for ListDirectory :

 void Main() { using (var client = new Renci.SshNet.SftpClient("sftp.host.com", "user", "password")) { var files = new List<String>(); client.Connect(); ListDirectory(client, ".", ref files); client.Disconnect(); files.Dump(); } } void ListDirectory(SftpClient client, String dirName, ref List<String> files) { foreach (var entry in client.ListDirectory(dirName)) { if (entry.IsDirectory) { ListDirectory(client, entry.FullName, ref files); } else { files.Add(entry.FullName); } } } 

Answer 2

Try the following:

 var filePaths = client.ListDirectory(client.WorkingDirectory); 

Answer 3

I achieved this using recursion. Created a TransportResponse class like this

  public class TransportResponse { public string directoryName { get; set; } public string fileName { get; set; } public DateTime fileTimeStamp { get; set; } public MemoryStream fileStream { get; set; } public List<TransportResponse> lstTransportResponse { get; set; } } 

I am creating a list of the TransportResponse class. If the directory_name is not null, it will contain a list of the same class as the files inside this directory in the form of a MemoryStream (this can be changed according to your use case)

 List<TransportResponse> lstResponse = new List<TransportResponse>(); using (var client = new SftpClient(connectionInfo)) { try { Console.WriteLine("Connecting to " + connectionInfo.Host + " ..."); client.Connect(); Console.WriteLine("Connected to " + connectionInfo.Host + " ..."); } catch (Exception ex) { Console.WriteLine("Could not connect to "+ connectionInfo.Host +" server. Exception Details: " + ex.Message); } if (client.IsConnected) { var files = client.ListDirectory(transport.SourceFolder); lstResponse = downloadFilesInDirectory(files, client); client.Disconnect(); } else { Console.WriteLine("Could not download files from "+ transport.TransportIdentifier +" because client was not connected."); } } private static List<TransportResponse> downloadFilesInDirectory(IEnumerable<SftpFile> files, SftpClient client) { List<TransportResponse> lstResponse = new List<TransportResponse>(); foreach (var file in files) { if (!file.IsDirectory) { if (file.Name != "." && file.Name != "..") { if (!TransportDAL.checkFileExists(file.Name, file.LastWriteTime)) { using (MemoryStream fs = new MemoryStream()) { try { Console.WriteLine("Reading " + file.Name + "..."); client.DownloadFile(file.FullName, fs); fs.Seek(0, SeekOrigin.Begin); lstResponse.Add(new TransportResponse { fileName = file.Name, fileTimeStamp = file.LastWriteTime, fileStream = new MemoryStream(fs.GetBuffer()) }); } catch(Exception ex) { Console.WriteLine("Error reading File. Exception Details: " + ex.Message); } } } else { Console.WriteLine("File was downloaded previously"); } } } else { if (file.Name != "." && file.Name != "..") { lstResponse.Add(new TransportResponse { directoryName = file.Name,lstTransportResponse = downloadFilesInDirectory(client.ListDirectory(file.Name), client) }); } } } return lstResponse; } 

Hope this helps. Thanks

Answer 4

@ Carlos Bos

There are some quirks in this library that make this recursive list complicated because the interaction between ChangeDirectory and ListDirectory doesn't work as you might expect.

right

It works great when the ChangeDirectory () parameter is "."

but if you do

 SftpClient sftp ...; sftp.ChangeDirectory("some_folder"); //get file list List<SftpFile> fileList = sftp.ListDirectory("some_folder").ToList(); 

that is, an assertion because the call to ListDirectory () expects "Some_folder / some_folder"

The workaround that I use is to save and restore the current directory before the remote download / renaming to "some_folder", and you need to specify this folder before the operation (for example, to see that the file already exists)

 string working_directory = sftp.WorkingDirectory; sftp.ChangeDirectory("some_folder"); sftp.RenameFile("name", "new_name"); sftp.ChangeDirectory(working_directory); 

to check if a file exists, this call is enough

 sftp.Exists(path) 

or if you want to add some other criteria, for example, case sensitive or not

  public FileExistence checkFileExists(string folder, string fileName) { //get file list List<SftpFile> fileList = sftp.ListDirectory(folder).ToList(); if (fileList == null) { return FileExistence.UNCONFIRMED; } foreach (SftpFile f in fileList) { Console.WriteLine(f.ToString()); //a not case sensitive comparison is made if (f.IsRegularFile && f.Name.ToLower() == fileName.ToLower()) { return FileExistence.EXISTS; } } //if not found in traversal , it does not exist return FileExistence.DOES_NOT_EXIST; } 

where FileExistence is

 public enum FileExistence { EXISTS, DOES_NOT_EXIST, UNCONFIRMED }; 

Answer 5

Here is the full class. This is a .NET Core 2.1 Http Trigger Application Application (v2)

I wanted to get rid of any directories starting with '.', Because on my sftp server there are .cache and .ssh folders with keys. Also did not want to deal with folder names such as '.' or ".."

Ultimately, I will project the SftpFile into the type I'm working with and return it to the caller (in this case, it will be a logic application). Then I will pass this object to the stored procedure and use OPENJSON to build my monitoring table. In fact, this is the first step in creating my SFTP processing queue, which will move the files from my SFTP folder to my data lake (for now, I think, until something better happens).

The reason I used .WorkingDirectory is because I created a user with a home directory like "/ home". This allows me to browse all my user folders. My application does not need to have a specific folder as a starting point, just the root user, so to speak.

 using Microsoft.AspNetCore.Http; using Microsoft.AspNetCore.Mvc; using Microsoft.Azure.WebJobs; using Microsoft.Azure.WebJobs.Extensions.Http; using Microsoft.Extensions.Logging; using Renci.SshNet; using Renci.SshNet.Sftp; using System; using System.Collections.Generic; using System.Linq; using System.Threading.Tasks; namespace SFTPFileMonitor { public class GetListOfFiles { [FunctionName("GetListOfFiles")] public async Task<IActionResult> RunAsync([HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequest req, ILogger log) { log.LogInformation("C# HTTP trigger function processed a request."); List<SftpFile> zFiles; int fileCount; decimal totalSizeGB; long totalSizeBytes; using (SftpClient sftpClient = new SftpClient("hostname", "username", "password")) { sftpClient.Connect(); zFiles = await GetFiles(sftpClient, sftpClient.WorkingDirectory, new List<SftpFile>()); fileCount = zFiles.Count; totalSizeBytes = zFiles.Sum(l => l.Length); totalSizeGB = BytesToGB(totalSizeBytes); } return new OkObjectResult(new { fileCount, totalSizeBytes, totalSizeGB, zFiles }); } private async Task<List<SftpFile>> GetFiles(SftpClient sftpClient, string directory, List<SftpFile> files) { foreach (SftpFile sftpFile in sftpClient.ListDirectory(directory)) { if (sftpFile.Name.StartsWith('.')) { continue; } if (sftpFile.IsDirectory) { await GetFiles(sftpClient, sftpFile.FullName, files); } else { files.Add(sftpFile); } } return files; } private decimal BytesToGB(long bytes) { return Convert.ToDecimal(bytes) / 1024 / 1024 / 1024; } } }