I am getting the following error on several, but not all, Vusers during a 1000 Vuser load test. Most but not all are crapping out at about the same iteration (23).

However, when I test and run the script from VUgen, it goes through all 73 values of {SearchTerm} with no errors.

Is there a way to see if this is a problem with controller or load generator? Again, some of the VUsers are able to run the entire 1000 Vuser 2 hour scenario with no errors.

Start auto log messages stack.
Ending iteration 22.
Starting iteration 23.
Error: Exception was raised when calling event-notify Vuser function in extension lrwreplaymain.dll: System Exceptions: EXCEPTION_ACCESS_VIOLATION
End auto log messages stack.

Start auto log messages stack - Iteration 23.
SearchSolutionsToConsider.c(9): Warning -26000: Evaluating script failed for "TD" DOM element (name="") "click", listed in the following line:  
SearchSolutionsToConsider.c(9): Warning -26000: ButtonClick('newSession');  
SearchSolutionsToConsider.c(9): web_element("New Session") highest severity level was "warning"  
SearchSolutionsToConsider.c(27): lr_think_time: 64.35 seconds (recorded think time was 56.00 seconds).
SearchSolutionsToConsider.c(29): web_browser("Verification_2") was successful  
SearchSolutionsToConsider.c(36): web_get_int_property was successful  
SearchSolutionsToConsider.c(49): Notify: Parameter Substitution: parameter "SearchTerm" =  "SIHD"
SearchSolutionsToConsider.c(49): Error: C interpreter run time error: SearchSolutionsToConsider.c (49):  Error -- memory violation : Exception ACCESS_VIOLATION received.
End auto log messages stack.

Start auto log messages stack - Iteration 23.
SearchSolutionsToConsider.c(49): Notify: CCI trace: SearchSolutionsToConsider.c(49): web_edit_field(0x044407a9 "textfield", 0x044407d5 "Snapshot=t4.inf", 0x044408b2 "DESCRIPTION", 0x0444079f "Type=text", 0x04440790 "Name=textfield", 0x04440777 "FrameName=searchBarFrame", 0x044408ab "ACTION", 0x04440761 "SetV.
SearchSolutionsToConsider.c(49): Notify: CCI trace: alue={SearchTerm}", 0x0444087d "LAST")
.
SearchSolutionsToConsider.c(49): Notify: CCI trace: Compiled_code(0): SearchSolutionsToConsider()
.
Action was aborted.
Ending Vuser...
Starting action vuser_end.
vuser_end.c(4): Error: C interpreter run time error: vuser_end.c (4):  Error -- memory violation : Exception ACCESS_VIOLATION received.
End auto log messages stack.

EDIT (Add Load Generator Info):
The breakdown of Vusers on load generators is as follows:
– 100 on a VM (low transaction rates doing other background tasks like browsing, etc.)
– 450 on a standalone system doing searches (not a VM)
– 450 on another standalone system doing searches (not a VM)

The 2 physical load generators that the 900 search Vusers are on are actually relatively new and more powerful machines built specifically for heavier load test scenarios. However, these are the systems and groups that we get the memory access errors on.

EDIT 2 – (11/18/2010: Add code from SearchSolutionsToConsider.c **Line 49 is the web_edit_field call. Formatting messed up the line numbers, but you can see from the log above what the action line numbers should be ):

    iRetVal = web_element("New Session",

    "Snapshot=t5.inf",

    DESCRIPTION,

    "Text=New Session",

    "Tag=SPAN",

    "FrameName=toolbarFrame",

    ACTION,

    "UserAction=Click",

    LAST);

if ((iHttpRetCodeIn == 404) || (iRetVal == 2)){

    lr_output_message("The function failed so exit the iteration.");

    lr_think_time(5); //Wait a few seconds before trying again

    lr_exit(LR_EXIT_MAIN_ITERATION_AND_CONTINUE ,LR_FAIL); //live to try another day

    }
lr_think_time(56); 
iRetVal = web_browser("Verificati