I'm writing something like a web crawler that it's engine follows these steps:
- Reading Rss Link(Argument)
- Defining a list(of) Rss Items
- Checking each link existence in database(SQL SERVER) by a separate query
If the link was new one it will insert the fields to DB by a separate query
Public Sub MyTickHandler() Dim NewItems As New List(Of Structures.RSSItem) Dim founded As Boolean = False NewItems = RssReader.ParseRssFile(RssURL) Dim connString = Configs.NewsDBConnection Dim myConnection As SqlConnection = New SqlConnection("Server=localhost;Database=db;Integrated Security=SSPI;;Connection Timeout=45;Max Pool Size= 300") myConnection.Open() For Each item In NewItems Dim cmdString As String = "SELECT id FROM posts with (nolock) WHERE link LIKE '" & item.link.Trim.ToLower & "'" Dim TheCommand As SqlCommand = New SqlCommand(cmdString, myConnection) Dim result = TheCommand.ExecuteScalar() If result Is Nothing Then TheCommand = New SqlCommand("INSERT INTO posts (link) VALUES ('" & item.link.ToLower.Trim & "')") TheCommand.Connection = myConnection TheCommand.ExecuteNonQuery() TheCommand = New SqlCommand("INSERT INTO queue (link,descrip,site,title,category) VALUES ('" & item.link.ToLower.Trim & "','" & StringToBase64(item.description) & "','" & RssSite & "','" & StringToBase64(item.title) & "','" & RssCategory & "')") TheCommand.Connection = myConnection TheCommand.ExecuteNonQuery() End If TheCommand.Dispose() Next myConnection.Close() myConnection.Dispose() SqlConnection.ClearPool(myConnection) End Sub
This works perfect for single calling.
but I have something about 150 Rss links and I should Check each of them every 2 minutes by threading, so by increasing the mount of SQL Queries, this process and also sql server won't response and application crashes!!
I tried some tips like increasing sql server response timeout, but it didn't help at all.
Any better way or tips for this process?
Thanks
LIKEquery that's killing performance. Run a SQL trace to find out. If that's not it, run the app under a profiler to find out where it spends its time.