Hello Mohammad,Alejandro

My job properties just specify the HDFS location of coordinator.xml

Mohammad, the directory for current(-2) was present in filesystem as I had already created the directory before running my job.

I am using Hadoop 0.21.0 here. Could this be a version incompatibility issue?

Thanks,
Rajat

--- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> , Mohammad Islam <kamrul@...> wrote:
>
>
> Hi Rajat,
> I believe the issue could be related to initial-instance ( initial-instance="${start}"). Because current(-2) will give empty directory for the first coordinator action.
> Please use different variable with value $start-3 hours as initial-instance.
>
> Check this:
> http://yahoo.github.com/oozie/releases/3.0.0/CoordinatorFunctionalSpec.html#a6.6.9._Dataset_Instance_Resolution_for_Instances_Before_the_Initial_Instance
>
> Regards,
> Mohammad
>
> On 5/5/11 6:55 AM, "Alejandro Abdelnur" <tucu@...> wrote:
>
>
>
>
>
>
> Rajat,
>
> What you are doing is correct, can you please check in the Oozie webconsole, the corresponding workflow job, what are the values of the job properties?
>
> Thanks.
>
> Alejandro
>
> On Thu, May 5, 2011 at 12:29 AM, rajatcs20 <rajatcs20@...> wrote:
> Hello,
>
> I have a use case where I want to send a property set in my coordinator.xml to my 'ssh' or 'Java' action in workflow.xml specification. How can this be done?
>
> My coordinator.xml file:
> <coordinator-app name="COORD_TEST" frequency="${coord:hours(1)}" start="${start}" end="${end}" timezone="${timezone}"
>                  xmlns="uri:oozie:coordinator:0.1">
>     <controls>
>         <timeout>10</timeout>
>         <concurrency>${concurrency_level}</concurrency>
>     </controls>
>     <datasets>
>         <dataset name="din" frequency="${coord:hours(1)}"
>                         initial-instance="${start}" timezone="${timezone}">
>                 <uri-template>hdfs://localhost:9000/user/root/input-data/${YEAR}${MONTH}/${DAY}/${HOUR}</uri-template>
>         </dataset>
>     </datasets>
>
>     <input-events>
>         <data-in name="in" dataset="din">
>                 <instance>${coord:current(-2)}</instance>
>         </data-in>
>     </input-events>
>     <action>
>         <workflow>
>             <app-path>hdfs://localhost:9000/user/root/mytest/coordinator</app-path>
>             <configuration>
>                 <property>
>                     <name>wInput</name>
>                     <value>${coord:dataIn('in')}</value>
>                 </property>
>             </configuration>
>         </workflow>
>     </action>
> </coordinator-app>
>
>
> My workflow.xml file:
> <workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
>   <start to='testjob'/>
>     <action name="testjob">
>         <ssh>
>             <host>192.168.0.128</host>
>             <command>/root/AtlasCubes/cubeCreator/binary/run-mapred.sh</command>
>             <args>${wInput}</args>
>         </ssh>
>         <ok to="end"/>
>         <error to="fail"/>
>     </action>
>     <kill name="fail">
>         <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
>     </kill>
>     <end name='end'/>
> </workflow-app>
>
> When I run the above, I see a runtime exception" Variable 'wInput' cannot be resolved.
>
> Can someone please help me? I want to use the value of 'wInput' in my script.
>
> Thanks,
> Rajat
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>

 
   

 
 
   

Hello Mohammad, Jens,

Thanks for your quick response. I already tried the instructions posted on the wiki page and it didn't work for me. I tried them with Hadoop 0.21.0. Is it possible that they don't work with this Hadoop release?

One more thing, is it mandatory to specify these two properties in job properties:
mapreduce.jobtracker.kerberos.principal=mapred/_HOST@LOCALHOST
dfs.namenode.kerberos.principal=hdfs/_HOST@LOCALHOST

Thanks,
Rajat

--- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> , Mohammad Islam <kamrul@...> wrote:
>
>
> Hi Jens,
> Thanks for your email and the proposal.
>
> So does the instruction work? If not, please let me know.
>
> Regards,
> Mohammad
>
> On 5/5/11 6:51 AM, "Jens Doerre" <doerre@...> wrote:
>
>
>
>
>
>
>    On 05/05/2011 09:21 AM, rajatcs20 wrote:
>
>
>
> Hello,
>
>
>
> Hi,
>
>
>
>
>
>
>  Does Oozie work with new Map Reduce APIs? I tried but I observe that reducer is not working properly. Same issue was seen by someone else on one of the mail threads. Has this issue been fixed or is there a workaround available on Hadoop 0.21.0?
>
>
>
>
>
> @Rajat
>  I have provided an example for using the new Hadoop API and tested it succesfully with CDHB3 Hadoop.
>  Mohammad has put it into the Wiki (and told the list about it):
>  https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
>
>  @Mohammad
>  I think it is hard to find.
>  Maybe the Wiki could be linked from the local documentation that comes with Oozie and also from the version hosted at Cloudera.
>
>
>
>
>
>
>
>  Thanks,
>  Rajat
>
>
>
>
>
> Greets,
>       Jens
>

 
   

Hello Mohammad,

I tried with data set initial instance < {start-3} but it didn't work.
With this change, workflow stays in WAITING state only. It doesn't run at all.

File coordinator.xml :
<coordinator-app name="COORD_TEST" frequency="${coord:hours(1)}"
start="${start}" end="${end}" timezone="${timezone}"
                  xmlns="uri:oozie:coordinator:0.1">
     <controls>
         <timeout>10</timeout>
         <concurrency>${concurrency_level}</concurrency>
     </controls>
     <datasets>
	 <dataset name="din" frequency="${coord:hours(1)}"
			 initial-instance="2011-05-09T10:00Z" timezone="${timezone}">
		
<uri-template>hdfs://localhost:9000/user/root/input-data/${YEAR}${MONTH}/${DAY}/\
${HOUR}</uri-template>
	 </dataset>
     </datasets>

     <input-events>
	 <data-in name="in" dataset="din">
		 <instance>${coord:current(-2)}</instance>
	 </data-in>
     </input-events>
     <action>
         <workflow>
            
<app-path>hdfs://localhost:9000/user/root/mytest/shellcmd/coordinator</app-path>
	     <configuration>
	         <property>
		     <name>wInput</name>
		     <value>${coord:dataIn('in')}</value>
	         </property>
	     </configuration>
         </workflow>
     </action>
</coordinator-app>

File coord-config.xml :
<configuration>
     <property>
         <name>app_path</name>
        
<value>hdfs://localhost:9000/user/root/mytest/shellcmd/coordinator</value>
     </property>
     <property>
         <name>language</name>
         <value>en</value>
     </property>
     <property>
         <name>country</name>
         <value>India</value>
     </property>
     <property>
         <name>start</name>
         <value>2011-05-09T16:22Z</value>
     </property>
     <property>
         <name>end</name>
         <value>2011-05-09T16:50Z</value>
     </property>
     <property>
         <name>timezone</name>
         <value>UTC</value>
     </property>
     <property>
         <name>timeout</name>
         <value>10</value>
     </property>
     <property>
         <name>concurrency_level</name>
         <value>2</value>
     </property>
     <property>
         <name>execution_order</name>
         <value>FIFO</value>
     </property>
     <property>
         <name>special_char</name>
         <value></value>
     </property>
     <!-- properties for workflow.xml -->
     <property>
         <name>jobTracker</name>
         <value>localhost:9001</value>
     </property>
     <property>
         <name>nameNode</name>
         <value>hdfs://localhost:9000</value>
     </property>
     <property>
	 <name>mapred.job.queue.name</name>
	 <value>default</value>
     </property>
</configuration>


Workflow.xml is same as in my first mail in the thread.
After the jon was started, the 'info' sub command showed output:
[root@station128 coordinator]# oozie job -info
0000006-110506205920921-oozie-root-C
Job ID : 0000006-110506205920921-oozie-root-C
--------------------------------------------------------------------------------\
----------------------------------------------------
Job Name : COORD_TEST
App Path : hdfs://localhost:9000/user/root/mytest/shellcmd/coordinator
Status   : SUCCEEDED
--------------------------------------------------------------------------------\
----------------------------------------------------
ID                                       Status    Ext ID
Err Code  Created          Nominal Time
0000006-110506205920921-oozie-root-C@1   WAITING   -
-         2011-05-09 16:20 2011-05-09 16:22
--------------------------------------------------------------------------------\
----------------------------------------------------

HDFS has data set instances for 2011-05-09T10:00Z, .... 2011-05-09T16:00Z

I can't figure out why the job is staying in WAITING state.

I ran the same job with Hadoop 0.20.2 and saw same behavior.

Thanks & Regards,
Rajat


--- In Oozie-users@yahoogroups.com, Mohammad Islam <kamrul@...> wrote:
>
>
> Rajat,
> Thanks for the info.
>
> I suspect the issue is not whether current(-2) is there or not.
> It is oozie coordinator that doesn't check for directory before
"initial-instance" .
>
> As also mentioned in the link "To address this edge scenario, Oozie
Coordinator silently ignores dataset instances out of bounds."
>
> Therefore, I asked to give earlier value (at least start - 3 hours) as dataset
initial-instance.
>
>
> Regards,
> Mohammad
>
>
>
> On 5/6/11 12:24 AM, "rajatcs20" <rajatcs20@...> wrote:
>
>
>
>
>
>
> Hello Mohammad,Alejandro
>
> My job properties just specify the HDFS location of coordinator.xml
>
> Mohammad, the directory for current(-2) was present in filesystem as I had
already created the directory before running my job.
>
> I am using Hadoop 0.21.0 here. Could this be a version incompatibility issue?
>
> Thanks,
> Rajat
>
> --- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> ,
Mohammad Islam <kamrul@> wrote:
> >
> >
> > Hi Rajat,
> > I believe the issue could be related to initial-instance (
initial-instance="${start}"). Because current(-2) will give empty directory for
the first coordinator action.
> > Please use different variable with value $start-3 hours as initial-instance.
> >
> > Check this:
> >
http://yahoo.github.com/oozie/releases/3.0.0/CoordinatorFunctionalSpec.html#a6.6\
.9._Dataset_Instance_Resolution_for_Instances_Before_the_Initial_Instance
> >
> > Regards,
> > Mohammad
> >
> > On 5/5/11 6:55 AM, "Alejandro Abdelnur" <tucu@> wrote:
> >
> >
> >
> >
> >
> >
> > Rajat,
> >
> > What you are doing is correct, can you please check in the Oozie webconsole,
the corresponding workflow job, what are the values of the job properties?
> >
> > Thanks.
> >
> > Alejandro
> >
> > On Thu, May 5, 2011 at 12:29 AM, rajatcs20 <rajatcs20@> wrote:
> > Hello,
> >
> > I have a use case where I want to send a property set in my coordinator.xml
to my 'ssh' or 'Java' action in workflow.xml specification. How can this be
done?
> >
> > My coordinator.xml file:
> > <coordinator-app name="COORD_TEST" frequency="${coord:hours(1)}"
start="${start}" end="${end}" timezone="${timezone}"
> >                  xmlns="uri:oozie:coordinator:0.1">
> >     <controls>
> >         <timeout>10</timeout>
> >         <concurrency>${concurrency_level}</concurrency>
> >     </controls>
> >     <datasets>
> >         <dataset name="din" frequency="${coord:hours(1)}"
> >                         initial-instance="${start}" timezone="${timezone}">
> >                
<uri-template>hdfs://localhost:9000/user/root/input-data/${YEAR}${MONTH}/${DAY}/\
${HOUR}</uri-template>
> >         </dataset>
> >     </datasets>
> >
> >     <input-events>
> >         <data-in name="in" dataset="din">
> >                 <instance>${coord:current(-2)}</instance>
> >         </data-in>
> >     </input-events>
> >     <action>
> >         <workflow>
> >            
<app-path>hdfs://localhost:9000/user/root/mytest/coordinator</app-path>
> >             <configuration>
> >                 <property>
> >                     <name>wInput</name>
> >                     <value>${coord:dataIn('in')}</value>
> >                 </property>
> >             </configuration>
> >         </workflow>
> >     </action>
> > </coordinator-app>
> >
> >
> > My workflow.xml file:
> > <workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
> >   <start to='testjob'/>
> >     <action name="testjob">
> >         <ssh>
> >             <host>192.168.0.128</host>
> >            
<command>/root/AtlasCubes/cubeCreator/binary/run-mapred.sh</command>
> >             <args>${wInput}</args>
> >         </ssh>
> >         <ok to="end"/>
> >         <error to="fail"/>
> >     </action>
> >     <kill name="fail">
> >         <message>Map/Reduce failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> >     </kill>
> >     <end name='end'/>
> > </workflow-app>
> >
> > When I run the above, I see a runtime exception" Variable 'wInput' cannot be
resolved.
> >
> > Can someone please help me? I want to use the value of 'wInput' in my
script.
> >
> > Thanks,
> > Rajat
> >
> >
> >
> > ------------------------------------
> >
> > Yahoo! Groups Links
> >
>

 
 
   

Hello Mohammad,

I tried with data set initial instance < {start-3} but it didn't work.
With this change, workflow stays in WAITING state only. It doesn't run at all.

File coordinator.xml :
<coordinator-app name="COORD_TEST" frequency="${coord:hours(1)}" start="${start}" end="${end}" timezone="${timezone}"
 xmlns="uri:oozie:coordinator:0.1">
 <controls>
 <timeout>10</timeout>
 <concurrency>${concurrency_level}</concurrency>
 </controls>
 <datasets>
<dataset name="din" frequency="${coord:hours(1)}"
initial-instance="2011-05-09T10:00Z" timezone="${timezone}">
<uri-template>hdfs://localhost:9000/user/root/input-data/${YEAR}${MONTH}/${DAY}/${HOUR}</uri-template>
</dataset>
 </datasets>

<input-events>
<data-in name="in" dataset="din">
<instance>${coord:current(-2)}</instance>
</data-in>
 </input-events>
 <action>
 <workflow>
 <app-path>hdfs://localhost:9000/user/root/mytest/shellcmd/coordinator</app-path>
    <configuration>
        <property>
    <name>wInput</name>
    <value>${coord:dataIn('in')}</value>
        </property>
    </configuration>
 </workflow>
 </action>
</coordinator-app>

File coord-config.xml :
<configuration>
 <property>
 <name>app_path</name>
 <value>hdfs://localhost:9000/user/root/mytest/shellcmd/coordinator</value>
 </property>
 <property>
 <name>language</name>
 <value>en</value>
 </property>
 <property>
 <name>country</name>
 <value>India</value>
 </property>
 <property>
 <name>start</name>
 <value>2011-05-09T16:22Z</value>
 </property>
 <property>
 <name>end</name>
 <value>2011-05-09T16:50Z</value>
 </property>
 <property>
 <name>timezone</name>
 <value>UTC</value>
 </property>
 <property>
 <name>timeout</name>
 <value>10</value>
 </property>
 <property>
 <name>concurrency_level</name>
 <value>2</value>
 </property>
 <property>
 <name>execution_order</name>
 <value>FIFO</value>
 </property>
 <property>
 <name>special_char</name>
 <value></value>
 </property>
 <!-- properties for workflow.xml -->
 <property>
 <name>jobTracker</name>
 <value>localhost:9001</value>
 </property>
 <property>
 <name>nameNode</name>
 <value>hdfs://localhost:9000</value>
 </property>
 <property>
<name>mapred.job.queue.name</name>
<value>default</value>
 </property>
</configuration>

Workflow.xml is same as in my first mail in the thread.
After the jon was started, the 'info' sub command showed output:
[root@station128 coordinator]# oozie job -info 0000006-110506205920921-oozie-root-C
Job ID : 0000006-110506205920921-oozie-root-C
----------------------------------------------------------
Job Name : COORD_TEST
App Path : hdfs://localhost:9000/user/root/mytest/shellcmd/coordinator
Status   : SUCCEEDED
----------------------------------------------------------
ID                                       Status    Ext ID                               Err Code  Created          Nominal Time     
0000006-110506205920921-oozie-root-C@1   WAITING   -                                    -         2011-05-09 16:20 2011-05-09 16:22
----------------------------------------------------------

HDFS has data set instances for 2011-05-09T10:00Z, .... 2011-05-09T16:00Z

I can't figure out why the job is staying in WAITING state.

I ran the same job with Hadoop 0.20.2 and saw same behavior.

Thanks & Regards,
Rajat

--- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> , Mohammad Islam <kamrul@...> wrote:
>
>
> Rajat,
> Thanks for the info.
>
> I suspect the issue is not whether current(-2) is there or not.
> It is oozie coordinator that doesn't check for directory before "initial-instance" .
>
> As also mentioned in the link "To address this edge scenario, Oozie Coordinator silently ignores dataset instances out of bounds."
>
> Therefore, I asked to give earlier value (at least start - 3 hours) as dataset initial-instance.
>
>
> Regards,
> Mohammad
>
>
>
> On 5/6/11 12:24 AM, "rajatcs20" <rajatcs20@...> wrote:
>
>
>
>
>
>
> Hello Mohammad,Alejandro
>
> My job properties just specify the HDFS location of coordinator.xml
>
> Mohammad, the directory for current(-2) was present in filesystem as I had already created the directory before running my job.
>
> I am using Hadoop 0.21.0 here. Could this be a version incompatibility issue?
>
> Thanks,
> Rajat
>
> --- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com>  <mailto:Oozie-users%40yahoogroups.com> , Mohammad Islam <kamrul@> wrote:
> >
> >
> > Hi Rajat,
> > I believe the issue could be related to initial-instance ( initial-instance="${start}"). Because current(-2) will give empty directory for the first coordinator action.
> > Please use different variable with value $start-3 hours as initial-instance.
> >
> > Check this:
> > http://yahoo.github.com/oozie/releases/3.0.0/CoordinatorFunctionalSpec.html#a6.6.9._Dataset_Instance_Resolution_for_Instances_Before_the_Initial_Instance
> >
> > Regards,
> > Mohammad
> >
> > On 5/5/11 6:55 AM, "Alejandro Abdelnur" <tucu@> wrote:
> >
> >
> >
> >
> >
> >
> > Rajat,
> >
> > What you are doing is correct, can you please check in the Oozie webconsole, the corresponding workflow job, what are the values of the job properties?
> >
> > Thanks.
> >
> > Alejandro
> >
> > On Thu, May 5, 2011 at 12:29 AM, rajatcs20 <rajatcs20@> wrote:
> > Hello,
> >
> > I have a use case where I want to send a property set in my coordinator.xml to my 'ssh' or 'Java' action in workflow.xml specification. How can this be done?
> >
> > My coordinator.xml file:
> > <coordinator-app name="COORD_TEST" frequency="${coord:hours(1)}" start="${start}" end="${end}" timezone="${timezone}"
> >                  xmlns="uri:oozie:coordinator:0.1">
> >     <controls>
> >         <timeout>10</timeout>
> >         <concurrency>${concurrency_level}</concurrency>
> >     </controls>
> >     <datasets>
> >         <dataset name="din" frequency="${coord:hours(1)}"
> >                         initial-instance="${start}" timezone="${timezone}">
> >                 <uri-template>hdfs://localhost:9000/user/root/input-data/${YEAR}${MONTH}/${DAY}/${HOUR}</uri-template>
> >         </dataset>
> >     </datasets>
> >
> >     <input-events>
> >         <data-in name="in" dataset="din">
> >                 <instance>${coord:current(-2)}</instance>
> >         </data-in>
> >     </input-events>
> >     <action>
> >         <workflow>
> >             <app-path>hdfs://localhost:9000/user/root/mytest/coordinator</app-path>
> >             <configuration>
> >                 <property>
> >                     <name>wInput</name>
> >                     <value>${coord:dataIn('in')}</value>
> >                 </property>
> >             </configuration>
> >         </workflow>
> >     </action>
> > </coordinator-app>
> >
> >
> > My workflow.xml file:
> > <workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
> >   <start to='testjob'/>
> >     <action name="testjob">
> >         <ssh>
> >             <host>192.168.0.128</host>
> >             <command>/root/AtlasCubes/cubeCreator/binary/run-mapred.sh</command>
> >             <args>${wInput}</args>
> >         </ssh>
> >         <ok to="end"/>
> >         <error to="fail"/>
> >     </action>
> >     <kill name="fail">
> >         <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> >     </kill>
> >     <end name='end'/>
> > </workflow-app>
> >
> > When I run the above, I see a runtime exception" Variable 'wInput' cannot be resolved.
> >
> > Can someone please help me? I want to use the value of 'wInput' in my script.
> >
> > Thanks,
> > Rajat
> >
> >
> >
> > ------------------------------------
> >
> > Yahoo! Groups Links
> >
>

 
   

Hi,

I have problems installing oozie, when start tomcat a have the error:
The web application [/oozie] registered the JBDC driver
[org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it when the
web application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.

Someone could help me?

Thanks

I'm using oozie 3.0.0, Hadoop 0.20.0  and pig 0.8.1, openjdk 6 and ubuntu 11.04,
tomcat 6.0.29.

I try with the file yahoo-oozie-3.0.0-0-g1c5d9b1.tar.gz made the wars and
installing in tomcat wepapp. Also using oozie-3.0.0-distro.tar.gz and the error
is the same.



--- In Oozie-users@yahoogroups.com, chao wang <brookwc@...> wrote:
>
> What version of Oozie are you installing?
> Also, for 2.3+, are you using embedded tomcat?
>
> I installed 2.3 and 3.0 using embedded tomcat with no problem at all.
>
> Regards,
>
> Chao
>
> On Mon, May 9, 2011 at 5:26 PM, Angel <agarciaz@...> wrote:
>
> >
> >
> > Hi,
> >
> > I have problems installing oozie, when start tomcat a have the error:
> > The web application [/oozie] registered the JBDC driver
> > [org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it when
> > the web application was stopped. To prevent a memory leak, the JDBC Driver
> > has been forcibly unregistered.
> >
> > Someone could help me?
> >
> > Thanks
> >
> >
> >
>

 
 
   

I'm using oozie 3.0.0, Hadoop 0.20.0  and pig 0.8.1, openjdk 6 and ubuntu 11.04, tomcat 6.0.29.

I try with the file yahoo-oozie-3.0.0-0-g1c5d9b1.tar.gz made the wars and installing in tomcat wepapp. Also using oozie-3.0.0-distro.tar.gz and the error is the same.

--- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> , chao wang <brookwc@...> wrote:
>
> What version of Oozie are you installing?
> Also, for 2.3+, are you using embedded tomcat?
>
> I installed 2.3 and 3.0 using embedded tomcat with no problem at all.
>
> Regards,
>
> Chao
>
> On Mon, May 9, 2011 at 5:26 PM, Angel <agarciaz@...> wrote:
>
> >
> >
> > Hi,
> >
> > I have problems installing oozie, when start tomcat a have the error:
> > The web application [/oozie] registered the JBDC driver
> > [org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it when
> > the web application was stopped. To prevent a memory leak, the JDBC Driver
> > has been forcibly unregistered.
> >
> > Someone could help me?
> >
> > Thanks
> >
> >  
> >
>

 
   

Thanks a lot Mohammad. Its working now :)

--- In Oozie-users@yahoogroups.com, Mohammad Islam <kamrul@...> wrote:
>
> Hi Rajat,
> I think we are getting in right direction.
>
> Do you think, you have the data directory that has _SUCCESS file?
>
> Run "oozie job –info <ID> -verbose". This will giving the missing directory as
"Missing dependency".
>
> According to your data set definition, you have to have "_SUCCESS" file.
However you could bypass that by specifying "<done-flag></done-flag>" in dataset
definition.
> Please look at the documentation of done-flag here:
>
http://yahoo.github.com/oozie/releases/3.0.0/CoordinatorFunctionalSpec.html#a5.1\
._Synchronous_Datasets
>
> Regards,
> Mohammad
>
>
>
>
>
>
> On 5/9/11 4:07 AM, "rajatcs20" <rajatcs20@...> wrote:
>
>
>
>
>
>
> Hello Mohammad,
>
> I tried with data set initial instance < {start-3} but it didn't work.
> With this change, workflow stays in WAITING state only. It doesn't run at all.
>
> File coordinator.xml :
> <coordinator-app name="COORD_TEST" frequency="${coord:hours(1)}"
start="${start}" end="${end}" timezone="${timezone}"
>  xmlns="uri:oozie:coordinator:0.1">
>  <controls>
>  <timeout>10</timeout>
>  <concurrency>${concurrency_level}</concurrency>
>  </controls>
>  <datasets>
> <dataset name="din" frequency="${coord:hours(1)}"
> initial-instance="2011-05-09T10:00Z" timezone="${timezone}">
>
<uri-template>hdfs://localhost:9000/user/root/input-data/${YEAR}${MONTH}/${DAY}/\
${HOUR}</uri-template>
> </dataset>
>  </datasets>
>
> <input-events>
> <data-in name="in" dataset="din">
> <instance>${coord:current(-2)}</instance>
> </data-in>
>  </input-events>
>  <action>
>  <workflow>
> 
<app-path>hdfs://localhost:9000/user/root/mytest/shellcmd/coordinator</app-path>
>     <configuration>
>         <property>
>     <name>wInput</name>
>     <value>${coord:dataIn('in')}</value>
>         </property>
>     </configuration>
>  </workflow>
>  </action>
> </coordinator-app>
>
> File coord-config.xml :
> <configuration>
>  <property>
>  <name>app_path</name>
>  <value>hdfs://localhost:9000/user/root/mytest/shellcmd/coordinator</value>
>  </property>
>  <property>
>  <name>language</name>
>  <value>en</value>
>  </property>
>  <property>
>  <name>country</name>
>  <value>India</value>
>  </property>
>  <property>
>  <name>start</name>
>  <value>2011-05-09T16:22Z</value>
>  </property>
>  <property>
>  <name>end</name>
>  <value>2011-05-09T16:50Z</value>
>  </property>
>  <property>
>  <name>timezone</name>
>  <value>UTC</value>
>  </property>
>  <property>
>  <name>timeout</name>
>  <value>10</value>
>  </property>
>  <property>
>  <name>concurrency_level</name>
>  <value>2</value>
>  </property>
>  <property>
>  <name>execution_order</name>
>  <value>FIFO</value>
>  </property>
>  <property>
>  <name>special_char</name>
>  <value></value>
>  </property>
>  <!-- properties for workflow.xml -->
>  <property>
>  <name>jobTracker</name>
>  <value>localhost:9001</value>
>  </property>
>  <property>
>  <name>nameNode</name>
>  <value>hdfs://localhost:9000</value>
>  </property>
>  <property>
> <name>mapred.job.queue.name</name>
> <value>default</value>
>  </property>
> </configuration>
>
> Workflow.xml is same as in my first mail in the thread.
> After the jon was started, the 'info' sub command showed output:
> [root@station128 coordinator]# oozie job -info
0000006-110506205920921-oozie-root-C
> Job ID : 0000006-110506205920921-oozie-root-C
> ----------------------------------------------------------
> Job Name : COORD_TEST
> App Path : hdfs://localhost:9000/user/root/mytest/shellcmd/coordinator
> Status   : SUCCEEDED
> ----------------------------------------------------------
> ID                                       Status    Ext ID
Err Code  Created          Nominal Time
> 0000006-110506205920921-oozie-root-C@1   WAITING   -
-         2011-05-09 16:20 2011-05-09 16:22
> ----------------------------------------------------------
>
> HDFS has data set instances for 2011-05-09T10:00Z, .... 2011-05-09T16:00Z
>
> I can't figure out why the job is staying in WAITING state.
>
> I ran the same job with Hadoop 0.20.2 and saw same behavior.
>
> Thanks & Regards,
> Rajat
>
> --- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> ,
Mohammad Islam <kamrul@> wrote:
> >
> >
> > Rajat,
> > Thanks for the info.
> >
> > I suspect the issue is not whether current(-2) is there or not.
> > It is oozie coordinator that doesn't check for directory before
"initial-instance" .
> >
> > As also mentioned in the link "To address this edge scenario, Oozie
Coordinator silently ignores dataset instances out of bounds."
> >
> > Therefore, I asked to give earlier value (at least start - 3 hours) as
dataset initial-instance.
> >
> >
> > Regards,
> > Mohammad
> >
> >
> >
> > On 5/6/11 12:24 AM, "rajatcs20" <rajatcs20@> wrote:
> >
> >
> >
> >
> >
> >
> > Hello Mohammad,Alejandro
> >
> > My job properties just specify the HDFS location of coordinator.xml
> >
> > Mohammad, the directory for current(-2) was present in filesystem as I had
already created the directory before running my job.
> >
> > I am using Hadoop 0.21.0 here. Could this be a version incompatibility
issue?
> >
> > Thanks,
> > Rajat
> >
> > --- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> 
<mailto:Oozie-users%40yahoogroups.com> , Mohammad Islam <kamrul@> wrote:
> > >
> > >
> > > Hi Rajat,
> > > I believe the issue could be related to initial-instance (
initial-instance="${start}"). Because current(-2) will give empty directory for
the first coordinator action.
> > > Please use different variable with value $start-3 hours as
initial-instance.
> > >
> > > Check this:
> > >
http://yahoo.github.com/oozie/releases/3.0.0/CoordinatorFunctionalSpec.html#a6.6\
.9._Dataset_Instance_Resolution_for_Instances_Before_the_Initial_Instance
> > >
> > > Regards,
> > > Mohammad
> > >
> > > On 5/5/11 6:55 AM, "Alejandro Abdelnur" <tucu@> wrote:
> > >
> > >
> > >
> > >
> > >
> > >
> > > Rajat,
> > >
> > > What you are doing is correct, can you please check in the Oozie
webconsole, the corresponding workflow job, what are the values of the job
properties?
> > >
> > > Thanks.
> > >
> > > Alejandro
> > >
> > > On Thu, May 5, 2011 at 12:29 AM, rajatcs20 <rajatcs20@> wrote:
> > > Hello,
> > >
> > > I have a use case where I want to send a property set in my
coordinator.xml to my 'ssh' or 'Java' action in workflow.xml specification. How
can this be done?
> > >
> > > My coordinator.xml file:
> > > <coordinator-app name="COORD_TEST" frequency="${coord:hours(1)}"
start="${start}" end="${end}" timezone="${timezone}"
> > >                  xmlns="uri:oozie:coordinator:0.1">
> > >     <controls>
> > >         <timeout>10</timeout>
> > >         <concurrency>${concurrency_level}</concurrency>
> > >     </controls>
> > >     <datasets>
> > >         <dataset name="din" frequency="${coord:hours(1)}"
> > >                         initial-instance="${start}"
timezone="${timezone}">
> > >                
<uri-template>hdfs://localhost:9000/user/root/input-data/${YEAR}${MONTH}/${DAY}/\
${HOUR}</uri-template>
> > >         </dataset>
> > >     </datasets>
> > >
> > >     <input-events>
> > >         <data-in name="in" dataset="din">
> > >                 <instance>${coord:current(-2)}</instance>
> > >         </data-in>
> > >     </input-events>
> > >     <action>
> > >         <workflow>
> > >            
<app-path>hdfs://localhost:9000/user/root/mytest/coordinator</app-path>
> > >             <configuration>
> > >                 <property>
> > >                     <name>wInput</name>
> > >                     <value>${coord:dataIn('in')}</value>
> > >                 </property>
> > >             </configuration>
> > >         </workflow>
> > >     </action>
> > > </coordinator-app>
> > >
> > >
> > > My workflow.xml file:
> > > <workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
> > >   <start to='testjob'/>
> > >     <action name="testjob">
> > >         <ssh>
> > >             <host>192.168.0.128</host>
> > >            
<command>/root/AtlasCubes/cubeCreator/binary/run-mapred.sh</command>
> > >             <args>${wInput}</args>
> > >         </ssh>
> > >         <ok to="end"/>
> > >         <error to="fail"/>
> > >     </action>
> > >     <kill name="fail">
> > >         <message>Map/Reduce failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > >     </kill>
> > >     <end name='end'/>
> > > </workflow-app>
> > >
> > > When I run the above, I see a runtime exception" Variable 'wInput' cannot
be resolved.
> > >
> > > Can someone please help me? I want to use the value of 'wInput' in my
script.
> > >
> > > Thanks,
> > > Rajat
> > >
> > >
> > >
> > > ------------------------------------
> > >
> > > Yahoo! Groups Links
> > >
> >
>

The error is from tomcat, in the log it write
"The web application [/oozie] registered the JBDC driver
[org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it when the
web application was stopped. To
prevent a memory leak, the JDBC Driver has been forcibly unregistered."

I don't know what's registered or unregister

Thanks

-------- Original Message --------
Subject: [SPAM] Re: [Oozie-users] Re: fail derby
From: Angelo Kaichen Huang <angeloh@...>
Date: Mon, May 09, 2011 6:27 pm
To: "Oozie-users@yahoogroups.com" <Oozie-users@yahoogroups.com>


What is the error message you got? What do you mean by registered or
unregistered?

Thanks,
Angelo


On 5/9/11 6:08 PM, "Angel" <agarciaz@...> wrote:



> 
> 
>   
>
>I'm using oozie 3.0.0, Hadoop 0.20.0  and pig 0.8.1, openjdk 6 and ubuntu
11.04, tomcat 6.0.29.
>
>I try with the file yahoo-oozie-3.0.0-0-g1c5d9b1.tar.gz made the wars
and installing in tomcat wepapp. Also using oozie-3.0.0-distro.tar.gz
and the error is the same.
>
>--- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> ,
chao wang <brookwc@...> wrote:
>>
>> What version of Oozie are you installing?
>> Also, for 2.3+, are you using embedded tomcat?
>>
>> I installed 2.3 and 3.0 using embedded tomcat with no problem at all.
>>
>> Regards,
>>
>> Chao
>>
>> On Mon, May 9, 2011 at 5:26 PM, Angel <agarciaz@...> wrote:
>>
>> >
>> >
>> > Hi,
>> >
>> > I have problems installing oozie, when start tomcat a have the error:
>> > The web application [/oozie] registered the JBDC driver
>> > [org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it when
>> > the web application was stopped. To prevent a memory leak, the JDBC Driver
>> > has been forcibly unregistered.
>> >
>> > Someone could help me?
>> >
>> > Thanks
>> >
>> >  
>> >
>>

Alejandro,

The problem is that tomcat don't start the oozie server, and only show the error
in the jdbc driver in the log.
I check the war file and is present the jdbc driver for derby, but it seems that
it don't load.

How I can change the derby's jdbc to mysql o hsqldb?



Thanks

-------- Original Message --------
Subject: [SPAM] Re: [Oozie-users] Re: fail derby
From: Alejandro Abdelnur <tucu@...>
Date: Tue, May 10, 2011 9:12 am
To: Oozie-users@yahoogroups.com


Angel,

JDBC drivers are registered before usage, that makes the JDBC layer aware of
their presence. What you are be seeing could be due to how Tomcat
finalizes classloaders when a webapp is stopped.

Is Oozie working and this problem happens only when you are shutting down Oozie?

Thanks.
Alejandro

On Tue, May 10, 2011 at 9:06 AM, Angel Garcia <agarciaz@...> wrote:
The error is from tomcat, in the log it write
"The web application [/oozie] registered the JBDC driver
[org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it
when the web application was stopped. To
prevent a memory leak, the JDBC Driver has been forcibly unregistered."

I don't know what's registered or unregister

Thanks

-------- Original Message --------
Subject: [SPAM] Re: [Oozie-users] Re: fail derby
From: Angelo Kaichen Huang <angeloh@...>
Date: Mon, May 09, 2011 6:27 pm
To: "Oozie-users@yahoogroups.com" <Oozie-users@yahoogroups.com>


What is the error message you got? What do you mean by registered or
unregistered?

Thanks,
Angelo


On 5/9/11 6:08 PM, "Angel" <agarciaz@...> wrote:



> 
> 
>   
>
>I'm using oozie 3.0.0, Hadoop 0.20.0  and pig 0.8.1, openjdk 6 and ubuntu
11.04, tomcat 6.0.29.
>
>I try with the file yahoo-oozie-3.0.0-0-g1c5d9b1.tar.gz made the wars
and installing in tomcat wepapp. Also using oozie-3.0.0-distro.tar.gz
and the error is the same.
>
>--- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> ,
chao wang <brookwc@...> wrote:
>>
>> What version of Oozie are you installing?
>> Also, for 2.3+, are you using embedded tomcat?
>>
>> I installed 2.3 and 3.0 using embedded tomcat with no problem at all.
>>
>> Regards,
>>
>> Chao
>>
>> On Mon, May 9, 2011 at 5:26 PM, Angel <agarciaz@...> wrote:
>>
>> >
>> >
>> > Hi,
>> >
>> > I have problems installing oozie, when start tomcat a have the error:
>> > The web application [/oozie] registered the JBDC driver
>> > [org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it when
>> > the web application was stopped. To prevent a memory leak, the JDBC Driver
>> > has been forcibly unregistered.
>> >
>> > Someone could help me?
>> >
>> > Thanks
>> >
>> >  
>> >
>>



 

Saludos Angel

Hey,

We have a lot of source code already written in cascading.
We would like to use oozie as a workflow engine using which we want to run
various tasks including the flows already written in cascading.

From the documentation it looks like oozie can be used only to run map reduce
jobs written directly using hadoop's java API, or using PIG.

Is it possible to disguise a map reduce job written using cascading as one of
these or as a single process java program, so that it can be one of the modules
in a larger workflow.

It does not make sense for us to change our source code and move away from
cascading.

Regards,

Hey,

Thanks for the quick reply.
If I am not mistaken, I read somewhere that the java action gets executed on any
one random machine on the cluster.

How will it take care that the cascading main class should run on the master
node in the hadoop cluster?

Regards,

--- In Oozie-users@yahoogroups.com, Alejandro Abdelnur <tucu@...> wrote:
>
> Harshal,
>
> You could have an Oozie Java action calling the cascading Main class. You'll
> have to have all Cascading JARs in the WF lib/ directory.
>
> Hope this helps.
>
> Alejandro
>
> On Tue, May 10, 2011 at 11:57 AM, harshal <harshal_022@...> wrote:
>
> > Hey,
> >
> > We have a lot of source code already written in cascading.
> > We would like to use oozie as a workflow engine using which we want to run
> > various tasks including the flows already written in cascading.
> >
> > From the documentation it looks like oozie can be used only to run map
> > reduce jobs written directly using hadoop's java API, or using PIG.
> >
> > Is it possible to disguise a map reduce job written using cascading as one
> > of these or as a single process java program, so that it can be one of the
> > modules in a larger workflow.
> >
> > It does not make sense for us to change our source code and move away from
> > cascading.
> >
> > Regards,
> >
> >
> >
> > ------------------------------------
> >
> > Yahoo! Groups Links
> >
> >
> >
> >
>

• Remove/replace existing properties
• Put your own value for each property.
• Make sure mysql jar is there in WEB-INF/lib.
• 
  

 
 
   

Alejandro,

The problem is that tomcat don't start the oozie server, and only show the error in the jdbc driver in the log.
I check the war file and is present the jdbc driver for derby, but it seems that it don't load.

How I can change the derby's jdbc to mysql o hsqldb?

Thanks

-------- Original Message --------
Subject: [SPAM] Re: [Oozie-users] Re: fail derby
From: Alejandro Abdelnur <tucu@... <mailto:tucu%40cloudera.com> >
Date: Tue, May 10, 2011 9:12 am
To: Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com>

Angel,

JDBC drivers are registered before usage, that makes the JDBC layer aware of their presence. What you are be seeing could be due to how Tomcat
finalizes classloaders when a webapp is stopped.

Is Oozie working and this problem happens only when you are shutting down Oozie?

Thanks.
Alejandro

On Tue, May 10, 2011 at 9:06 AM, Angel Garcia <agarciaz@... <mailto:agarciaz%40yahoo.com> > wrote:
The error is from tomcat, in the log it write
"The web application [/oozie] registered the JBDC driver
[org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it
when the web application was stopped. To
prevent a memory leak, the JDBC Driver has been forcibly unregistered."

I don't know what's registered or unregister

Thanks

-------- Original Message --------
Subject: [SPAM] Re: [Oozie-users] Re: fail derby
From: Angelo Kaichen Huang <angeloh@... <mailto:angeloh%40yahoo-inc.com> >
Date: Mon, May 09, 2011 6:27 pm
To: "Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> " <Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> >

What is the error message you got? What do you mean by registered or unregistered?

Thanks,
Angelo

On 5/9/11 6:08 PM, "Angel" <agarciaz@... <mailto:agarciaz%40yahoo.com> > wrote:

>
>
>   
>
>I'm using oozie 3.0.0, Hadoop 0.20.0  and pig 0.8.1, openjdk 6 and ubuntu 11.04, tomcat 6.0.29.
>
>I try with the file yahoo-oozie-3.0.0-0-g1c5d9b1.tar.gz made the wars
and installing in tomcat wepapp. Also using oozie-3.0.0-distro.tar.gz
and the error is the same.
>
>--- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com>  <mailto:Oozie-users%40yahoogroups.com> , chao wang <brookwc@...> wrote:
>>
>> What version of Oozie are you installing?
>> Also, for 2.3+, are you using embedded tomcat?
>>
>> I installed 2.3 and 3.0 using embedded tomcat with no problem at all.
>>
>> Regards,
>>
>> Chao
>>
>> On Mon, May 9, 2011 at 5:26 PM, Angel <agarciaz@...> wrote:
>>
>> >
>> >
>> > Hi,
>> >
>> > I have problems installing oozie, when start tomcat a have the error:
>> > The web application [/oozie] registered the JBDC driver
>> > [org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it when
>> > the web application was stopped. To prevent a memory leak, the JDBC Driver
>> > has been forcibly unregistered.
>> >
>> > Someone could help me?
>> >
>> > Thanks
>> >
>> >  
>> >
>>

 

Saludos Angel
 
   

 
 
   


Hi Angel,
Could you try the following setting into oozie-site.xml?

You need to do/verify these too:

• Remove/replace existing properties
• Put your own value for each property.
• Make sure mysql jar is there in WEB-INF/lib.
• 
  

<property>
        <name>oozie.service.DataSourceService.jdbc.driver</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>
            JDBC driver class.
        </description>
    </property>

    <property>
        <name>oozie.service. StoreService.jdbc.url</name>
        <value>jdbc:mysql://localhost:3306</value>
        <description>
            JDBC URL.
        </description>
    </property>

    <property>
        <name>oozie.service. StoreService.jdbc.username</name>
        <value>oozie</value>
        <description>
            DB user name.
        </description>
    </property>

    <property>
        <name>oozie.service. StoreService.jdbc.password</name>
        <value>oozie</value>
        <description>
            DB user password.
            IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value, if empty Configuration
                       assumes it is NULL.
        </description>
    </property>


Regards,
Mohammad

On 5/10/11 11:19 AM, "Angel Garcia" <agarciaz@...> wrote:

 
 
   

Alejandro,

The problem is that tomcat don't start the oozie server, and only show the error in the jdbc driver in the log.
I check the war file and is present the jdbc driver for derby, but it seems that it don't load.

How I can change the derby's jdbc to mysql o hsqldb?

Thanks

-------- Original Message --------
Subject: [SPAM] Re: [Oozie-users] Re: fail derby
From: Alejandro Abdelnur <tucu@... <mailto:tucu%40cloudera.com> >
Date: Tue, May 10, 2011 9:12 am
To: Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com>

Angel,

JDBC drivers are registered before usage, that makes the JDBC layer aware of their presence. What you are be seeing could be due to how Tomcat
finalizes classloaders when a webapp is stopped.

Is Oozie working and this problem happens only when you are shutting down Oozie?

Thanks.
Alejandro

On Tue, May 10, 2011 at 9:06 AM, Angel Garcia <agarciaz@... <mailto:agarciaz%40yahoo.com> > wrote:
The error is from tomcat, in the log it write
"The web application [/oozie] registered the JBDC driver
[org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it
when the web application was stopped. To
prevent a memory leak, the JDBC Driver has been forcibly unregistered."

I don't know what's registered or unregister

Thanks

-------- Original Message --------
Subject: [SPAM] Re: [Oozie-users] Re: fail derby
From: Angelo Kaichen Huang <angeloh@... <mailto:angeloh%40yahoo-inc.com> >
Date: Mon, May 09, 2011 6:27 pm
To: "Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> " <Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com> >

What is the error message you got? What do you mean by registered or unregistered?

Thanks,
Angelo

On 5/9/11 6:08 PM, "Angel" <agarciaz@... <mailto:agarciaz%40yahoo.com> > wrote:

>
>
>   
>
>I'm using oozie 3.0.0, Hadoop 0.20.0  and pig 0.8.1, openjdk 6 and ubuntu 11.04, tomcat 6.0.29.
>
>I try with the file yahoo-oozie-3.0.0-0-g1c5d9b1.tar.gz made the wars
and installing in tomcat wepapp. Also using oozie-3.0.0-distro.tar.gz
and the error is the same.
>
>--- In Oozie-users@yahoogroups.com <mailto:Oozie-users%40yahoogroups.com>  <mailto:Oozie-users%40yahoogroups.com> , chao wang <brookwc@...> wrote:
>>
>> What version of Oozie are you installing?
>> Also, for 2.3+, are you using embedded tomcat?
>>
>> I installed 2.3 and 3.0 using embedded tomcat with no problem at all.
>>
>> Regards,
>>
>> Chao
>>
>> On Mon, May 9, 2011 at 5:26 PM, Angel <agarciaz@...> wrote:
>>
>> >
>> >
>> > Hi,
>> >
>> > I have problems installing oozie, when start tomcat a have the error:
>> > The web application [/oozie] registered the JBDC driver
>> > [org.apache.derby.jdbc.AutoloadedDriver] but failed to unregister it when
>> > the web application was stopped. To prevent a memory leak, the JDBC Driver
>> > has been forcibly unregistered.
>> >
>> > Someone could help me?
>> >
>> > Thanks
>> >
>> >  
>> >
>>

 

Saludos Angel
 
   

   

Hi,

I am facing an issue while running hourly (coordinator) jobs.
I had a backlog of 12 hours.
After I ran the coordinator what I see is that each job is getting
invoked one after another having a time period of 5 minutes.
Even though I had set the concurrency to 4.
Is there a way to reduce the time period of invocation from 5 min to a
lower value.
And I did not get why 4 job were not getting launched at the same time.

(In between I also had a backlog of minute jobs but I had no issue with
them.)


Regards,
Jaydeep



________________________________

The information contained in this communication is intended solely for the use
of the individual or entity to whom it is addressed and others authorized to
receive it. It may contain confidential or legally privileged information. If
you are not the intended recipient you are hereby notified that any disclosure,
copying, distribution or taking any action in reliance on the contents of this
information is strictly prohibited and may be unlawful. If you have received
this communication in error, please notify us immediately by responding to this
email and then delete it from your system. The firm is neither liable for the
proper and complete transmission of the information contained in this
communication nor for any delay in its receipt.

 
 
   

Hi,

I am facing an issue while running hourly (coordinator) jobs.
I had a backlog of 12 hours.
After I ran the coordinator what I see is that each job is getting
invoked one after another having a time period of 5 minutes.
Even though I had set the concurrency to 4.
Is there a way to reduce the time period of invocation from 5 min to a
lower value.
And I did not get why 4 job were not getting launched at the same time.

(In between I also had a backlog of minute jobs but I had no issue with
them.)

Regards,
Jaydeep

________________________________

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
 
   

 
 
   

HI All

oozie starts generated log files in Gb's within few hours. I am not running a single job. I checked the log files and it has these line in it.


2011-05-12 10:00:00,079  WARN CallableQueueService:528 - USER[-] GROUP[-] max concurrency for callable [coord_action_input] exceeded, requeueing with [500]ms delay
2011-05-12 10:00:00,079  WARN CallableQueueService:528 - USER[-] GROUP[-] max concurrency for callable [coord_action_input] exceeded, requeueing with [500]ms delay
2011-05-12 10:00:00,079  WARN CallableQueueService:528 - USER[-] GROUP[-] max concurrency for callable [coord_action_input] exceeded, requeueing with [500]ms delay

I am not sure what happen to oozie and why its start giving this message in log files.

Any idea what happening?

 

Hi Jaydeep,
What version of oozie?

How many coordinator actions are in WAITING state? SQL like “Select count(*) from COORD_ACTIONS where status = ‘WAITING’” could give a number.

Did the input dependencies meet? For any instance of hourly job, could you check how many coord actions are in state READY for that specific job?

What is the size of oozie internal queue?
>From UI, select “Instrumentation” tab then select “Samplers/CallableQueue/Queue.size””.

Regards,
Mohammad


 

On 5/12/11 6:02 AM, "jaydeep vishwakarma" <jaydeep.vishwakarma@...> wrote:

 
 
   

Hi,

I am facing an issue while running hourly (coordinator) jobs.
I had a backlog of 12 hours.
After I ran the coordinator what I see is that each job is getting
invoked one after another having a time period of 5 minutes.
Even though I had set the concurrency to 4.
Is there a way to reduce the time period of invocation from 5 min to a
lower value.
And I did not get why 4 job were not getting launched at the same time.

(In between I also had a backlog of minute jobs but I had no issue with
them.)

Regards,
Jaydeep

________________________________

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
 
   

 
 
   

no, i do not have any active job in the system. i had killed all the active jobs. Also i had restart the oozie many times and eveny restart the system also. but every time it starts, its start making the log.
  

On Thu, May 12, 2011 at 11:18 PM, Mohammad Islam <kamrul@...> wrote:

 
 
 
   

Hi Shuja,
Do you have any active job in the system? I’m sure you have some jobs and that are filling up the oozie queue.
You could kill those jobs and restart the oozie server should bring down the log size.

What version of oozie?

Regards,
Mohammad




On 5/12/11 10:46 AM, "Shuja Rehman" <shujamughal@... <http://shujamughal@...> > wrote:

 
 
   

HI All

oozie starts generated log files in Gb's within few hours. I am not running a single job. I checked the log files and it has these line in it.


2011-05-12 10:00:00,079  WARN CallableQueueService:528 - USER[-] GROUP[-] max concurrency for callable [coord_action_input] exceeded, requeueing with [500]ms delay
2011-05-12 10:00:00,079  WARN CallableQueueService:528 - USER[-] GROUP[-] max concurrency for callable [coord_action_input] exceeded, requeueing with [500]ms delay
2011-05-12 10:00:00,079  WARN CallableQueueService:528 - USER[-] GROUP[-] max concurrency for callable [coord_action_input] exceeded, requeueing with [500]ms delay

I am not sure what happen to oozie and why its start giving this message in log files.

Any idea what happening?

 
   
 

Hi Mohammad,

Here are the requested details.

1. oozie version : 2.3.0-cdh3uO
1. There were no coordinators in waiting state.
2. All Input dependency were there. No job was in READY state. I was getting only RUNNING state, But after every 5 minutes.
3. “Samplers/CallableQueue/Queue.size” size is 246.5833 .

Regards,
Jaydeep

On 05/12/2011 11:10 PM, Mohammad Islam wrote:

Hi Jaydeep,
What version of oozie?

How many coordinator actions are in WAITING state? SQL like “Select count(*) from COORD_ACTIONS where status = ‘WAITING’” could give a number.

Did the input dependencies meet? For any instance of hourly job, could you check how many coord actions are in state READY for that specific job?

What is the size of oozie internal queue?
>From UI, select “Instrumentation” tab then select “Samplers/CallableQueue/Queue.size””.

Regards,
Mohammad


 

On 5/12/11 6:02 AM, "jaydeep vishwakarma" <jaydeep.vishwakarma@...> wrote:

 
 
   

Hi,

I am facing an issue while running hourly (coordinator) jobs.
I had a backlog of 12 hours.
After I ran the coordinator what I see is that each job is getting
invoked one after another having a time period of 5 minutes.
Even though I had set the concurrency to 4.
Is there a way to reduce the time period of invocation from 5 min to a
lower value.
And I did not get why 4 job were not getting launched at the same time.

(In between I also had a backlog of minute jobs but I had no issue with
them.)

Regards,
Jaydeep

________________________________

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
 
   

---

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.

 
 
   

Hi Jaydeep,
I’m little confused that there is no coordinator action (not job!) in either ready or waiting state. Looks like they are in RUNNING state. Is it correct? Did you check COORD_ACTIONS table?

How many actions are there for that hourly job?
You could run this: “SELECT COUNT( * ), STATUS from COORD_ACTIONs where job_id = 'XXXXX-XXXXXX-YYYY-C' group by STATUS;”

Regards,
Mohammad


On 5/12/11 11:31 AM, "jaydeep vishwakarma" <jaydeep.vishwakarma@...> wrote:

Hi Mohammad,

Here are the requested details.

1. oozie version : 2.3.0-cdh3uO
1. There were no coordinators in waiting state.
2. All Input dependency were there. No job was in READY state. I was getting only RUNNING state, But after every 5 minutes.
3. “Samplers/CallableQueue/Queue.size” size is 246.5833 .

Regards,
Jaydeep

On 05/12/2011 11:10 PM, Mohammad Islam wrote:

Hi Jaydeep,
What version of oozie?

How many coordinator actions are in WAITING state? SQL like “Select count(*) from COORD_ACTIONS where status = ‘WAITING’” could give a number.

Did the input dependencies meet? For any instance of hourly job, could you check how many coord actions are in state READY for that specific job?

What is the size of oozie internal queue?
>From UI, select “Instrumentation” tab then select “Samplers/CallableQueue/Queue.size””.

Regards,
Mohammad


 

On 5/12/11 6:02 AM, "jaydeep vishwakarma" <jaydeep.vishwakarma@...> wrote:

 
 
   

Hi,

I am facing an issue while running hourly (coordinator) jobs.
I had a backlog of 12 hours.
After I ran the coordinator what I see is that each job is getting
invoked one after another having a time period of 5 minutes.
Even though I had set the concurrency to 4.
Is there a way to reduce the time period of invocation from 5 min to a
lower value.
And I did not get why 4 job were not getting launched at the same time.

(In between I also had a backlog of minute jobs but I had no issue with
them.)

Regards,
Jaydeep

________________________________

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
 
   

---

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.

   

>I am facing an issue while running hourly (coordinator) jobs.
>I had a backlog of 12 hours.
>After I ran the coordinator what I see is that each job is getting
>invoked one after another having a time period of 5 minutes.
>Even though I had set the concurrency to 4.
>Is there a way to reduce the time period of invocation from 5 min to a
>lower value.
>And I did not get why 4 job were not getting launched at the same time.


This is expected behavior. Oozie default materializes (creates) the actions for next hour of coordinator job. If you have 5 minutes frequency in coordinator job, you should see 12 actions are created every 5 minutes. If you have 15 minutes frequency in coordinator job, you should see 4 actions are created every 5 minutes. If you have one hour frequency  in coord job, you should see 1 action is created every 5 minutes.

The 5 minutes you saw is the interval of the materialization service to kick off the materialization process. You can minimize the interval of the materialization service to less than default 5 minutes by putting smaller value of “oozie.service.CoordMaterializeTriggerService.lookup.interval” in oozie-site.xml. You also can adjust the value of materialization window to larger value like 2 hours or something.

    <property>
       <name>oozie.service.CoordMaterializeTriggerService.lookup.interval
        </name>
        <value>300</value>
        <description> Coordinator Job Lookup trigger command is scheduled at
            this "interval" (in seconds).</description>
    </property>

    <property>
       <name>oozie.service.CoordMaterializeTriggerService.materialization.window
        </name>
        <value>3600</value>
        <description> Coordinator Job Lookup command materialized each job for
            this next "window" duration</description>
    </property>


Thanks,
Angelo

On 5/12/11 1:47 PM, "Mohammad Islam" <kamrul@...> wrote:

 
 
   

Hi Jaydeep,
I’m little confused that there is no coordinator action (not job!) in either ready or waiting state. Looks like they are in RUNNING state. Is it correct? Did you check COORD_ACTIONS table?

How many actions are there for that hourly job?
You could run this: “SELECT COUNT( * ), STATUS from COORD_ACTIONs where job_id = 'XXXXX-XXXXXX-YYYY-C' group by STATUS;”

Regards,
Mohammad


On 5/12/11 11:31 AM, "jaydeep vishwakarma" <jaydeep.vishwakarma@...> wrote:

Hi Mohammad,

Here are the requested details.

1. oozie version : 2.3.0-cdh3uO
1. There were no coordinators in waiting state.
2. All Input dependency were there. No job was in READY state. I was getting only RUNNING state, But after every 5 minutes.
3. “Samplers/CallableQueue/Queue.size” size is 246.5833 .

Regards,
Jaydeep

On 05/12/2011 11:10 PM, Mohammad Islam wrote:

Hi Jaydeep,
What version of oozie?

How many coordinator actions are in WAITING state? SQL like “Select count(*) from COORD_ACTIONS where status = ‘WAITING’” could give a number.

Did the input dependencies meet? For any instance of hourly job, could you check how many coord actions are in state READY for that specific job?

What is the size of oozie internal queue?
>From UI, select “Instrumentation” tab then select “Samplers/CallableQueue/Queue.size””.

Regards,
Mohammad


 

On 5/12/11 6:02 AM, "jaydeep vishwakarma" <jaydeep.vishwakarma@...> wrote:

 
 
   

Hi,

I am facing an issue while running hourly (coordinator) jobs.
I had a backlog of 12 hours.
After I ran the coordinator what I see is that each job is getting
invoked one after another having a time period of 5 minutes.
Even though I had set the concurrency to 4.
Is there a way to reduce the time period of invocation from 5 min to a
lower value.
And I did not get why 4 job were not getting launched at the same time.

(In between I also had a backlog of minute jobs but I had no issue with
them.)

Regards,
Jaydeep

________________________________

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
 
   

---

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.