Quantcast
Channel: AutoIt v3 - General Help and Support
Viewing all articles
Browse latest Browse all 12506

Comparing Large Files using SQLite (Not sure I'm doing this right)

$
0
0

Hi all!

 

I have two files. One of which, I read into a SQLite database and the other I read line by line and search the SQLite database for the lines from the file. It works, but it's really slow, so I'm wondering if it's my process or just the nature of the beast.

 

File2 (FSTEST.dat) is 8mb and File1 (ESTEST.dat) is 1.5mb. These are the smaller of the sets of files (the other sets can be upwards of 50mb or so).

 

Is there a way to make this run faster? Thanks for the help as always!

AutoIt         
#include <File.au3> #include <SQLite.au3> #include <SQLite.dll.au3> Global $hQuery, $aRow, $fl_array2 Global $sTempDB = @ScriptDir & "\TempDB.db" Global $iTotalCount = 0, $iTotalSoFar = 0 Global $iKey1_a = 1 Global $iKey1_b = 12 Global $iKey2_a = 18 Global $iKey2_b = 25 Global $st = TimerInit() Global $iOrphans, $iDupes, $iMatches ; --------------------------------------------- ; Start SQLite ; --------------------------------------------- _SQLite_Startup() If @error Then  ConsoleWrite('error loading sqlite.dll' & @LF)  Exit EndIf If FileExists($sTempDB) Then FileDelete($sTempDB) _SQLite_Open($sTempDB) If @error Then  ConsoleWrite('Unable to open DB' & @LF)  _Exit() EndIf ; --------------------------------------------- ; Get the files to compare ; --------------------------------------------- $sFile1 = FileOpenDialog("Select EXTEST.DAT", @ScriptDir, "Dat files (*.dat)", 1 + 2, "") If @error Then _Exit() $sFile2 = FileOpenDialog("Select FSTEST.DAT", @ScriptDir, "Dat files (*.dat)", 1 + 2, "") If @error Then _Exit() ; --------------------------------------------- ; Load the data from File2 into the db ; --------------------------------------------- _ReLoad() ; --------------------------------------------- ; Compare the fields from File1 to those in File2 ; --------------------------------------------- CompareItems($sFile1) ; --------------------------------------------- Func CompareItems($sFileContents)  Dim $aSplit  _FileReadToArray($sFileContents, $aSplit)  If IsArray($aSplit) Then   If $aSplit[1] = "" Then    _ArrayDelete($aSplit, 1)    $aSplit[0] -= 1   EndIf   $iTotalCount += $aSplit[0]   For $i = 1 To $aSplit[0]    $sPart1 = StringMid($aSplit[$i], $iKey1_a, ($iKey1_b - $iKey1_a) + 1)    $sPart2 = StringMid($aSplit[$i], $iKey2_a, ($iKey2_b - $iKey2_a) + 1)    If $sPart1 <> "" Then     $iCount = 0     _SQLite_Query(-1, 'select * from compare where PART1 = "' & $sPart1 & '" and PART2 = "' & $sPart2 & '";', $hQuery)     While _SQLite_FetchData($hQuery, $aRow) = $SQLite_OK      If $aRow[0] <> "" Then $iCount += 1     WEnd     ConsoleWrite($sPart1 & " $iCount = " & $iCount & @CRLF)     If $iCount = 0 And $i <> $aSplit[0] Then      $iOrphans += 1     EndIf     If $iCount = 1 Then      $iMatches += 1     EndIf     If $iCount > 1 Then      $iDupes += 1     EndIf    EndIf    $iTotalSoFar += 1   Next   ConsoleWrite("---------------------------" & @CRLF)   ConsoleWrite("Orphans: " & $iOrphans & @CRLF)   ConsoleWrite("Matches: " & $iMatches & @CRLF)   ConsoleWrite("Dupes: " & $iDupes & @CRLF)   ConsoleWrite("Total: " & $iTotalCount & @CRLF)  EndIf EndFunc   ;==>CompareItems Func _ReLoad()  ; drop "parts" table if it exists, re-define and reload it  If _SQLite_Exec(-1, 'drop table if exists compare;') <> $SQLite_OK Then   ConsoleWrite('Drop table failed' & @LF)   _Exit()  Else   ConsoleWrite('Compare table dropped for refresh' & @LF)  EndIf  If _SQLite_Exec(-1, 'create table compare (PART1, PART2);') <> $SQLite_OK Then   ConsoleWrite('Create Table Failed' & @LF)   _Exit()  EndIf  _FileReadToArray($sFile2, $fl_array2)  Switch @error   Case 1    ConsoleWrite('Input file failed to open' & @LF)    _Exit()   Case 2    ConsoleWrite('Unable to split file' & @LF)    _Exit()  EndSwitch  Local $aLine, $sql  ProgressOn('Loading Compare Table', 'Please Wait', '', Default, Default, 2 + 16); 2 - without always on top / 16 - Window can be moved  _SQLite_Exec(-1, "begin immediate;")  For $1 = 1 To $fl_array2[0]   ProgressSet(($1 / $fl_array2[0]) * 100)   $sql = 'insert into compare values ("' & _     StringMid($fl_array2[$1], $iKey1_a, ($iKey1_b - $iKey1_a) + 1) & '", "' & _     StringMid($fl_array2[$1], $iKey2_a, ($iKey2_b - $iKey2_a) + 1) & '");'   If _SQLite_Exec(-1, $sql) <> $SQLite_OK Then    ConsoleWrite('Table insert failed STMT = ' & $sql & @LF)    _Exit()   EndIf  Next  _SQLite_Exec(-1, "commit;")  ProgressOff()  ConsoleWrite('Table loaded with ' & UBound($fl_array2) - 1 & ' records in ' & Round(TimerDiff($st) / 1000, 3) & ' seconds' & @LF) EndFunc   ;==>_ReLoad Func _Exit()  _SQLite_Close()  _SQLite_Shutdown()  Exit EndFunc   ;==>_Exit

(I don't know why the code box always loses my tab structure...weird)

Attached Files


Viewing all articles
Browse latest Browse all 12506

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>