OS Security & Endpoint Protection

Requirements: Monitor all file operations with minimal performance impact. Architecture: (1) Minifilter driver registered with Filter Manager at appropriate altitude (320000-329999 for anti-virus). (2) Register pre-operation callbacks for IRP_MJ_CREATE, IRP_MJ_WRITE, IRP_MJ_SET_INFORMATION (rename/delete). (3) In pre-create: check file hash against blocklist, scan if needed. For performance: maintain an in-memory cache of clean file hashes, only scan on first access or after write. (4) Post-operation callbacks for write operations: mark file as dirty (needs re-scan). (5) Communication with user-mode service via filter communication port (FltCreateCommunicationPort). (6) User-mode service handles: heavy scanning (AV engine), cloud lookups, logging. Performance: Use asynchronous scanning for non-executable files, synchronous only for PE/script files. Maintain per-file state using stream contexts.

Signals to monitor: (1) Process created in suspended state (CREATE_SUSPENDED flag). (2) NtUnmapViewOfSection called on the new process (unmapping original code). (3) VirtualAllocEx + WriteProcessMemory to the target (injecting new code). (4) SetThreadContext to change entry point. (5) ResumeThread to start execution. Detection logic: Correlate these events in sequence within a short time window. Additional checks: Compare the on-disk image path (what the process says it is) with the actual memory content (hash of in-memory code vs file on disk). If they don't match — hollowing detected. Defender approach: Uses kernel callbacks to catch process creation + ETW for memory operations, then behavioral ML model scores the sequence.

Mechanism: (1) Minifilter driver intercepts all IRP_MJ_CREATE/WRITE operations targeting protected folders. (2) For each write attempt, identifies the requesting process (via PID from I/O request). (3) Checks process against an allowlist (maintained in registry and updated from cloud). (4) If process is NOT on allowlist: block the write (STATUS_ACCESS_DENIED), log the event, notify user. (5) System processes and known-good apps (Office, etc.) are pre-approved. Challenges: False positives (legitimate new apps blocked), user experience (must manually approve new apps), and performance (checking allowlist on every write to protected folders). Ransomware impact: Even if ransomware is running with admin privileges, it can't modify protected folders because the minifilter operates at the I/O manager level before the filesystem.

How AMSI works: (1) Script hosts (PowerShell, VBScript, JScript, Office macros) call AmsiScanBuffer before executing content. (2) AMSI passes content to registered anti-malware provider (Defender). (3) Provider scans and returns clean/malware verdict. (4) Script host proceeds or blocks based on verdict. Bypass attempts: (1) Patching amsi.dll in memory (overwrite AmsiScanBuffer to return clean). (2) Unhooking — restore original bytes of hooked functions. (3) Reflection — use .NET reflection to set amsiInitFailed = true. (4) Obfuscation — encode payload to evade string-based detection. (5) Running in a context where AMSI isn't loaded. Counter-defenses: ETW monitoring of amsi.dll modifications, code integrity checks, behavioral detection of the bypass attempt itself.

Data sources: (1) Windows Security Event Logs — logon events (4624 type 3/10), explicit credentials (4648). (2) Process creation events (4688) — PsExec, WMIC, PowerShell remoting patterns. (3) Network connections — SMB (445), WinRM (5985/5986), RDP (3389) between workstations. (4) Kerberos ticket requests — TGS for unusual services. Detection logic: Build a graph of normal communication patterns. Alert on: (a) First-time connections between hosts, (b) Admin tools (PsExec, WMIC) from non-admin workstations, © Service account used from unexpected sources, (d) Rapid sequential connections (A→B→C→D in minutes). Architecture: Stream processing (Kafka + Flink), graph database for relationship storage, ML model for anomaly scoring based on historical patterns.