Swarm/DysonNetwork.Drive/Index

File Indexing System Documentation

Overview

The File Indexing System provides a hierarchical file organization layer on top of the existing file storage system in DysonNetwork Drive. It allows users to organize their files in folders and paths while maintaining the underlying file storage capabilities.

When using with the gateway, replace the /api with the /drive in the path. And all the arguments will be transformed into snake case via the gateway.

Architecture

Core Components

1. SnCloudFileIndex Model - Represents the file-to-path mapping
2. FileIndexService - Business logic for file index operations
3. FileIndexController - REST API endpoints for file management
4. FileUploadController Integration - Automatic index creation during upload

Database Schema

-- File Indexes table
CREATE TABLE "FileIndexes" (
    "Id" uuid NOT NULL DEFAULT gen_random_uuid(),
    "Path" character varying(8192) NOT NULL,
    "FileId" uuid NOT NULL,
    "AccountId" uuid NOT NULL,
    "CreatedAt" timestamp with time zone NOT NULL DEFAULT (now() at time zone 'utc'),
    "UpdatedAt" timestamp with time zone NOT NULL DEFAULT (now() at time zone 'utc'),
    CONSTRAINT "PK_FileIndexes" PRIMARY KEY ("Id"),
    CONSTRAINT "FK_FileIndexes_Files_FileId" FOREIGN KEY ("FileId") REFERENCES "Files" ("Id") ON DELETE CASCADE,
    INDEX "IX_FileIndexes_Path_AccountId" ("Path", "AccountId")
);

API Endpoints

Browse Files

GET /api/index/browse?path=/documents/

Browse files in a specific path.

Query Parameters:

• path (optional, default: "/") - The path to browse

Response:

{
  "path": "/documents/",
  "files": [
    {
      "id": "guid",
      "path": "/documents/",
      "fileId": "guid",
      "accountId": "guid",
      "createdAt": "2024-01-01T00:00:00Z",
      "updatedAt": "2024-01-01T00:00:00Z",
      "file": {
        "id": "string",
        "name": "document.pdf",
        "size": 1024,
        "mimeType": "application/pdf",
        "hash": "sha256-hash",
        "uploadedAt": "2024-01-01T00:00:00Z",
        "expiredAt": null,
        "hasCompression": false,
        "hasThumbnail": true,
        "isEncrypted": false,
        "description": null
      }
    }
  ],
  "totalCount": 1
}

Get All Files

GET /api/index/all

Get all files for the current user across all paths.

Response:

{
  "files": [
    // Same structure as browse endpoint
  ],
  "totalCount": 10
}

Move File

POST /api/index/move/{indexId}

Move a file to a new path.

Path Parameters:

• indexId - The file index ID

Request Body:

{
  "newPath": "/archived/"
}

Response:

{
  "fileId": "guid",
  "indexId": "guid",
  "oldPath": "/documents/",
  "newPath": "/archived/",
  "message": "File moved successfully"
}

Remove File Index

DELETE /api/index/remove/{indexId}?deleteFile=false

Remove a file index. Optionally delete the actual file data.

Path Parameters:

• indexId - The file index ID

Query Parameters:

• deleteFile (optional, default: false) - Whether to also delete the file data

Response:

{
  "message": "File index removed successfully",
  "fileId": "guid",
  "fileName": "document.pdf",
  "path": "/documents/",
  "fileDataDeleted": false
}

Clear Path

DELETE /api/index/clear-path?path=/temp/&deleteFiles=false

Remove all file indexes in a specific path.

Query Parameters:

• path (optional, default: "/") - The path to clear
• deleteFiles (optional, default: false) - Whether to also delete orphaned files

Response:

{
  "message": "Cleared 5 file indexes from path",
  "path": "/temp/",
  "removedCount": 5,
  "filesDeleted": false
}

Create File Index

POST /api/index/create

Create a new file index for an existing file.

Request Body:

{
  "fileId": "guid",
  "path": "/documents/"
}

Response:

{
  "indexId": "guid",
  "fileId": "guid",
  "path": "/documents/",
  "message": "File index created successfully"
}

Search Files

GET /api/index/search?query=report&path=/documents/

Search for files by name or metadata.

Query Parameters:

• query (required) - The search query
• path (optional) - Limit search to specific path

Response:

{
  "query": "report",
  "path": "/documents/",
  "results": [
    // Same structure as browse endpoint
  ],
  "totalCount": 3
}

Path Normalization

The system automatically normalizes paths to ensure consistency:

• Trailing Slash: All paths end with /
• Root Path: User home folder is represented as /
• Query Safety: Paths are validated to avoid SQL injection
• Examples:
  • /documents/ ✅ (correct)
  • /documents → /documents/ ✅ (normalized)
  • /documents/reports/ ✅ (correct)
  • /documents/reports → /documents/reports/ ✅ (normalized)

File Upload Integration

When uploading files with the FileUploadController, you can specify a path to automatically create file indexes:

Create Upload Task Request:

{
  "fileName": "document.pdf",
  "fileSize": 1024,
  "contentType": "application/pdf",
  "hash": "sha256-hash",
  "path": "/documents/"  // New field for file indexing
}

The system will automatically create a file index when the upload completes successfully.

Service Methods

FileIndexService

public class FileIndexService
{
    // Create a new file index
    Task<SnCloudFileIndex> CreateAsync(string path, Guid fileId, Guid accountId);
    
    // Get files by path
    Task<List<SnCloudFileIndex>> GetByPathAsync(Guid accountId, string path);
    
    // Get all files for account
    Task<List<SnCloudFileIndex>> GetByAccountIdAsync(Guid accountId);
    
    // Get indexes for specific file
    Task<List<SnCloudFileIndex>> GetByFileIdAsync(Guid fileId);
    
    // Move file to new path
    Task<SnCloudFileIndex?> UpdateAsync(Guid indexId, string newPath);
    
    // Remove file index
    Task<bool> RemoveAsync(Guid indexId);
    
    // Remove all indexes in path
    Task<int> RemoveByPathAsync(Guid accountId, string path);
    
    // Normalize path format
    public static string NormalizePath(string path);
}

Error Handling

The API returns appropriate HTTP status codes and error messages:

• 400 Bad Request: Invalid input parameters
• 401 Unauthorized: User not authenticated
• 403 Forbidden: User lacks permission
• 404 Not Found: Resource not found
• 500 Internal Server Error: Server-side error

Error Response Format:

{
  "code": "BROWSE_FAILED",
  "message": "Failed to browse files",
  "status": 500
}

Security Considerations

1. Ownership Verification: All operations verify that the user owns the file indexes
2. Path Validation: Paths are normalized and validated
3. Cascade Deletion: File indexes are automatically removed when files are deleted
4. Safe File Deletion: Files are only deleted when no other indexes reference them

Usage Examples

Upload File to Specific Path

# Create upload task with path
curl -X POST /api/files/upload/create \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{
    "fileName": "report.pdf",
    "fileSize": 2048,
    "contentType": "application/pdf",
    "path": "/documents/reports/"
  }'

Browse Files

curl -X GET "/api/index/browse?path=/documents/reports/" \
  -H "Authorization: Bearer {token}"

Move File

curl -X POST "/api/index/move/{indexId}" \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{"newPath": "/archived/"}'

Search Files

curl -X GET "/api/index/search?query=invoice&path=/documents/" \
  -H "Authorization: Bearer {token}"

Best Practices

1. Use Trailing Slashes: Always include trailing slashes in paths
2. Organize Hierarchically: Use meaningful folder structures
3. Search Efficiently: Use the search endpoint instead of client-side filtering
4. Clean Up: Use the clear-path endpoint for temporary directories
5. Monitor Usage: Check total file counts for quota management

Integration Notes

• The file indexing system works alongside the existing file storage
• Files can exist in multiple paths (hard links)
• File deletion is optional and only removes data when safe
• The system maintains referential integrity between files and indexes