import csv error using SQL Loader and perl

Question

Hello all I have a question I hope you guys can help me with. I tried to include all the relevant info. I'm building a perl script that will eventually loop though different sqlloader control files and import their respective csv data into oracle sql database tables. I'm testing multiple control loads before looping them. The problem is that I get an error even though the script connects to the db and uploads all the csv data without any problems that I can see. all the rows are accounted for and the log doesn't really help:

 ================================================================================
 [root@sanasr06 scripts]# perl db_upload.pl
 connection made! Starting database upload...
 Error: Can't open import control_general to SQL DB : at db_upload.pl line 44
 ================================================================================

line 44 is the system connection:

system ("sqlldr $userid\@$sid/$passwd control=@control_pools log=$log silent=all ") or $logger->logdie("Error: Can't open import control data to SQL DB :$!");

I'm including the control file output, the perl script and the control file. (the skipped file mentioned is for the csv headers:)

 SQL*Loader: Release 11.2.0.1.0 - Production on Tue Aug 14 12:32:36 2012
 Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
 Control File:   /despliegue/san/project/sql_ctrl/general.ctl
 Character Set UTF8 specified for all input.

 Data File:      /despliegue/san/project/csv/Pools.csv
 Bad File:     /despliegue/san/project/logs/sql_error.bad
 Discard File: /despliegue/san/project/logs/sql_discard.dsc 
 (Allow all discards)

 Number to load: ALL
 Number to skip: 1
 Errors allowed: 50
 Bind array:     64 rows, maximum of 256000 bytes
 Continuation:    none specified
 Path used:      Conventional
 Silent options: FEEDBACK, ERRORS and DISCARDS

 Table I_GENERAL, loaded from every logical record.
 Insert option in effect for this table: TRUNCATE
 TRAILING NULLCOLS option in effect
 Column Name                  Position   Len  Term Encl Datatype
 ------------------------------ ---------- ----- ---- ---- ---------------------
 OBJECTID   (FILLER FIELD)           FIRST     *     ,  O(")  CHARACTER

 DESCRIPTION   (FILLER FIELD)         NEXT     *     ,  O(")  CHARACTER  

 SERIALNUMBER                         NEXT     *     ,  O(")  CHARACTER  

 PRODUCT_NAME                         NEXT     *     ,  O(")  CHARACTER   

 CONTROLLER_VERSION                   NEXT     *     ,  O(")  CHARACTER    

 NUMBER_OF_CONTROLLERS                NEXT     *     ,  O(")  CHARACTER    

 CAPACITY_GB                          NEXT     *     ,  O(")  CHARACTER    

 PRODUCT_CODE                         NEXT     *     ,  O(")  CHARACTER            

 value used for ROWS parameter changed from 64 to 15

 Table I_GENERAL:
 2512 Rows successfully loaded.
 0 Rows not loaded due to data errors.
 0 Rows not loaded because all WHEN clauses were failed.
 0 Rows not loaded because all fields were null.

 Space allocated for bind array:                 247680 bytes(15 rows)
 Read   buffer bytes: 1048576
 Total logical records skipped:          1
 Total logical records read:          2512
 Total logical records rejected:         0
 Total logical records discarded:        0

 Run began on Tue Aug 14 12:32:36 2012
 Run ended on Tue Aug 14 12:32:38 2012

==================================================================================

the above file is of course shortened but includes all the relevant information. here's the perl script.

#!/usr/bin/perl
use strict;
use warnings;
use DBI;
use Log::Log4perl;
#this script loads multiple saved csv files into the database using the control files

################ Initialization #############################################
my $homepath = "/despliegue/san/project";

my $log_conf = "$homepath/logs/log.conf";
Log::Log4perl->init($log_conf)or die("Error: Can't open log.config Does it exist? $!");
my $logger = Log::Log4perl->get_logger();

################ database connection variables####
my ($serial, $model);
my $host="me.notyou33.safety";
my $port="1426";
my $userid="user";
my $passwd="pass";
my $sid="sid";
my $log="$homepath/logs/sql_import.log";

#Control file location
my @control_pools= "$homepath/sql_ctrl/pools.ctl";
my @control_general = "$homepath/sql_ctrl/general.ctl";
my @control_ports= "$homepath/sql_ctrl/ports.ctl";
my @control_replication = "$homepath/sql_ctrl/replication.ctl";

#######################Database connection and data upload #################
my $dbh = DBI->connect( "dbi:Oracle:host=$host;sid=$sid;port=$port", "$userid", "$passwd", 
{ RaiseError => 1}) or $logger->logdie ("Database connection not made: $DBI::errstr");

print " connection made! Starting database upload...\n";

system ("sqlldr $userid\@$sid/$passwd control=@control_general log=$log silent=all") or $logger->logdie("Error: Can't open import control_general to SQL DB :$!");

print "one done moving to next one\n";

system ("sqlldr $userid\@$sid/$passwd control=@control_pools log=$log silent=all ") or $logger->logdie("Error: Can't open import control data to SQL DB :$!");

system ("sqlldr $userid\@$sid/$passwd control=@control_ports log=$log  ") or $logger->logdie("Error: Can't open import control data to SQL DB :$!");

print "three done moving to last one\n";

system ("sqlldr $userid\@$sid/$passwd control=@control_replication log=$log silent=feedback ") or    $logger->logdie("Error: Can't open import control data to SQL DB :$!");

print "................Done\n";
############################################################################
$dbh->disconnect;

================================================================================== the control file:

OPTIONS (SKIP=1) 
LOAD DATA
CHARACTERSET UTF8
INFILE      '/despliegue/san/project/csv/Pools.csv'
BADFILE     '/despliegue/san/project/logs/sql_error.bad'
DISCARDFILE '/despliegue/san/project/logs/sql_discard.dsc'
TRUNCATE INTO TABLE I_GENERAL 
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY "\"" 
 TRAILING NULLCOLS
( 
OBJECTID FILLER,
DESCRIPTION FILLER,
SERIALNUMBER,
PRODUCT_NAME,
CONTROLLER_VERSION,
NUMBER_OF_CONTROLLERS,
CAPACITY_GB,
PRODUCT_CODE,
)

tauli · Accepted Answer · 2012-08-14 12:07:21Z

1

system() returns the return value of the wait call which includes the return value of the program you executed. If everything goes right, this will be 0. This is different from almost all other funktions in Perl, where you expect them to return some Value which evaluates to True in boolean context. Therefore, the commonly used error handling by using the or operator, does not work properly. You might want to try something like this instead:

system ("sqlldr $userid\@$sid/$passwd control=@control_pools log=$log silent=all") == 0
    or $logger->logdie("Error: Can't open import control data to SQL DB :$?");

You can read more about handling the return value of system() in the documentation under perldoc -f system

answered Aug 14, 2012 at 12:07

tauli

1,4208 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Comocho00 Over a year ago

That was it! Thanks a lot Tauli. Im gonna read more about the system call now!

w.k · Accepted Answer · 2012-08-14 12:00:50Z

1

There is Logdie which should be logdie, AFAIU

answered Aug 14, 2012 at 12:00

w.k

8,4765 gold badges36 silver badges64 bronze badges

1 Comment

Comocho00 Over a year ago

Thanks but that was a typo. Fixed!

tbone · Accepted Answer · 2012-08-14 12:26:07Z

1

The problem is the system call expects a return value of 0 to be "successful". Your sqlldr job, if it skips or discards a record, will not return 0 (I've seen it return 2, check docs to be sure). So, unless you load all records successfully, your perl script (as written) will exit out.

perl system
sqlldr return codes

answered Aug 14, 2012 at 12:26

tbone

15.5k3 gold badges38 silver badges42 bronze badges

3 Comments

Comocho00 Over a year ago

I guess I'm too much of a noob. I Need 15 reputation, sorry!

tbone Over a year ago

@Comocho00 hmmm, you seem to have 18 ;-)

Comocho00 Over a year ago

Hey it worked lol. I DID try it when you mentioned it though. Thanks again!

user491135 · Accepted Answer · 2012-08-14 12:47:26Z

1

In my case I execute the sqlldr with backticks (similar to system), that helps me to get any feedback in a variable.

my $sqlldr = "sqlldr userid=usr/pss\@TNS control=\'$controlfile\' log=\'$logfile\' silent=header,feedback";
$execution = `$sqlldr 2>&1`;

The trick is that the returned value is not 0 in perl and you have to shift right 8 bits that value to be sure that you get 0. In my case I do as follows:

# Get the returned code from the last execution
my $ret = $? >> 8;

if ($ret == 0) {
    $logger->info("Class DLA_Upload: All rows were successfully loaded");
}
elsif ($ret == 1) {
    die("Class DLA_Upload: Executing sqlldr returned the following error:\n$execution");
}
elsif ($ret == 2) {
    $logger->info("Class DLA_Upload: SQL*Loader was executed but some or all rows were rejected or discarded, please check $logfile for further information");
}
else {
    die("Class DLA_Upload: FATAL ERROR: sqlldr corrupted or not found");
}

Why, here you have a link from Perl monks that explains it properly.

answered Aug 14, 2012 at 12:47

user491135

2 Comments

Comocho00 Over a year ago

Thanks that looks interesting too, I like the way you generated the errors, similar to the return code example below. I'm looking into it now!

user491135 Over a year ago

Thanks @Comocho00, the thing is that sqlldr has some exit codes (they have them documented) depending how was the upload, and in my case is very important to know if everything was uploaded or we had any rejects

Collectives™ on Stack Overflow

import csv error using SQL Loader and perl

4 Answers 4

1 Comment

1 Comment

3 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

1 Comment

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related