Spark runs in local mode - ksfzhaohui's personal page - OSCHINA - Chinese open source technology exchange community

Spark local mode operation

original

Blogs before 2016

2015/12/19 12:23

Reading number 10K

one Spark Introduction
Spark is a distributed computing based on the map reduce algorithm, which has the advantages of Hadoop MapReduce; However, unlike MapReduce, the intermediate output and results of the job can be saved in memory, thus no need to read or write HDFS. Therefore, Spark can be better applied to map reduce algorithms requiring iteration such as data mining and machine learning.

2. Operation mode
Local mode
Standalone mode
Mesoes mode
Yarn mode

Let's test it“ Local mode ”The running condition of this test is Windows7

1. Environmental preparation

jdk 1.6
Spark 1.2.0
Official website: http://spark.apache.org/downloads.html
Or: http://archive.apache.org/dist/spark/
We can download the source code [spark 1.2.0. tgz] and compile it by ourselves, or directly download [spark 1.2.0 bin hadoop2.4. tgz]
Note: 1.6 is installed in the local jdk. Spark1.5+can only run on jdk7,8;
Scala 2.11.7 http://www.scala-lang.org/download/
Python 2.7.7 https://www.python.org/downloads/windows/

2. Contents

Will download【 spark-1.2.0-bin-hadoop2.4.tgz 】Decompress

Bin: a command to facilitate our local mode testing
Conf: Some configuration templates. We can remove the template from xxxx.template and reconfigure it

By default, it directly supports scala and Python running, and executes respectively: spark shell.cmd and pyspark. cmd

function spark-shell.cmd：

function pyspark.cmd：

1. Of course, Spark itself supports the java language. We can execute java programs through Spark-submit.cmd

Create a maven project to test:

 <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.2.0</version> </dependency>

It will automatically download all the dependent jar packages. There are many jar packages, so you need to wait a little bit

 package com.spark; import java.util.Arrays; import java.util.regex.Pattern; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.function.FlatMapFunction; public class WordCount { private static final Pattern SPACE = Pattern.compile(" "); public static void main(String[] args) { SparkConf conf = new SparkConf().setAppName("JavaWordCount"); JavaSparkContext ctx = new JavaSparkContext(conf); String filePath = "D:/systemInfo.log"; JavaRDD<String> lines = ctx.textFile(filePath, 1); JavaRDD<String> words = lines .flatMap(new FlatMapFunction<String, String>() { @Override public Iterable<String> call(String s) { return Arrays.asList(SPACE.split(s)); } }); System.out.println("wordCount:" + words.count()); } }

Make our project into a jar package through maven

Spark submit parameter description: Usage: spark-submit [options] <app jar | python file> [app options]

See this for details: http://my.oschina.net/u/140462/blog/519409

You can view the relevant information of Spark through port 4040: http://localhost:4040

2. Java programs can also be used without Spark submit can be run locally:

 package com.spark; import java.io.BufferedReader; import java.io.InputStreamReader; import java.util.Arrays; import java.util.regex.Pattern; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.function.FlatMapFunction; public final class JavaWordCount { private static final Pattern SPACE = Pattern.compile(" "); public static void main(String[] args) throws Exception { SparkConf conf = new SparkConf().setMaster("local").setAppName( "JavaWordCount"); JavaSparkContext ctx = new JavaSparkContext(conf); String filePath = ""; BufferedReader reader = new BufferedReader(new InputStreamReader( System.in)); System.out.println("Enter FilePath:"); System.out.println("e.g. D:/systemInfo.log"); while (true) { System.out.println("> "); filePath = reader.readLine(); if (filePath.equalsIgnoreCase("exit")) { ctx.stop(); } else { JavaRDD<String> lines = ctx.textFile(filePath, 1); JavaRDD<String> words = lines .flatMap(new FlatMapFunction<String, String>() { @Override public Iterable<String> call(String s) { return Arrays.asList(SPACE.split(s)); } }); System.out.println("wordCount:" + words.count()); } } } }

The results are as follows:

 Enter FilePath: e.g. D:/systemInfo.log >  D:/systemInfo.log wordCount:48050 >

The setMaster is local. The specific master URL parameters are shown in the following figure:

Reference above: http://www.infoq.com/cn/articles/apache-spark-introduction

Bright 2024-05-19 23:25

What a fool! I killed myself. How can people deal with me later.

Hakuna 2024-05-31 18:28

It is compatible with Oracle, but does not know "just" or "just". Those who can be compatible with Oracle and do well are real men and real warriors. You should know that compatibility means that even bugs must be compatible, and you have no other code that can not be copied. It's all based on real skills and understanding of oracle.

hanf 2024-05-31 17:45

If only the design and architecture are similar, what's the point? Good things must be learned, and you can't prove that the design is not the same. As for the source code, you also said that neither Oracle nor Damon is open source, and you can't prove it. There are many people who question Dream, but so far, no one has come up with strong evidence. You should at least provide evidence to copy

zhy 2024-05-16 13:16

At the end of Shannon is Nong

Love to eat raw pears 2024-06-01 19:18

Don't expect programmers to have a deep understanding of the document. I still think that since the tool hides the details of $#, some necessary security checks are necessary. Many people do not use MybatisPlus directly, but use various so-called rapid development platforms. The MyBatisPlus rapid development platform Snowy, Guns, etc., has an impression that many versions have the problem of using Wrapper directly to splice the Request parameter. I remember that JeecgBoot was opened a lot of CVEs last year or the year before last because of the Wrapper splicing problem. Do you know the author of ibeetl? Many CVE blaming holes have been opened before. The problem is similar. The lack of basic knowledge "script editing permission" is actively handed over to the front end. What a low-level error or even low-energy behavior. However, I accepted it with an open mind and added a white list check.

Qin Liming 2024-05-11 09:12

be devoid of any sense of shame

Yeah, for 2024-05-17 13:42

That's too right. Old Zhou can't control Google, but he can control 360. Do not do to others what you do not want. All 360 products should be opened first.

People are addicted to food 2024-06-01 13:53

History history combination

Ning Jinnong 2024-06-01 21:04

Correct it. The example of loading the library is wrong. It should be # library=@ loading the dynamic library, "./yards to the treasurer. dll"

Yokesily 2024-06-02 15:11

So designed

One code Yma 2024-05-06 09:14

My technical article was moved by CSDN. Why didn't anyone step on the sewing machine? This kind of report is a joke to me. The monsters with background are fine, and the monsters without background fight to death

Starry Night Destiny 2024-06-01 21:49

It feels like Mybatis. It's OK to provide users with optional security solutions. It's useless for users to complain about this problem

Shen Lang Panda 2024-06-01 08:16

You can directly ask questions in the project work order. The comment area is not suitable for answering such questions

One code Yma 2024-05-09 09:58

Recently, I often go to interviews. People who hate Ali background most regard me as a fool, even though I am a fool

Voice of God 2024-06-01 20:47

By default, injection ($) and splicing are turned off. If you want to use it, you need to sign the birth and death form and press the fingerprint.

osc_92224065 2024-04-29 10:57

Long term oppressed outsourcing of state-owned enterprises

Rocket ship 2024-05-31 19:22

It's a ghost anyway.

Love to eat raw pears 2024-06-01 11:48

Why is this so-called "vulnerability" not a vulnerability? Spring, MyBatis and other frameworks can accept all kinds of CVE criticism, while MyBatisPlus has to dump the pot and accuse programmers of being too low-level# There is a difference. The premise is that you write XML, MyBatisPlus encapsulates Wrapper and claims to simplify code. Since it encapsulates and hides $#, it is not appropriate to do some necessary security checks? Instead of doubting the authority of CVE, you should know that SQL ->MyBatis ->MyBatisPlus ->various back-end scaffolds have multiple layers, each layer is simplifying, and each layer is throwing away the upper layer of the boiler. Who dares to use them. The programmers who use MyBatisPlus can't be expected to be at a high level. Every programmer wants to save effort. The front-end parameters can be directly obtained by HttpServletRequest from the back-end. Wrapper splicing can be found everywhere. If something goes wrong, is it the front-end or the framework? According to Qingmiao, can the injection vulnerability of the previous log4j and the deletion vulnerability of the Druid be used to eliminate low-level programmers?

Dogo_Little People 2024-06-02 12:24

Not everyone will go to see the document in full detail. As a general basic framework, the method naming should consider not only readability but also understandability. At least, it should also establish a cognition for developers. LambdaQueryWrapper is recommended. The official only briefly said that QueryWrapper may lead to SQL injection risks, There are no detailed examples (many people don't understand what SQL injection is). Now I met a jerk and submitted it to CVE to see who is the most powerful

Code craftsman 2024-06-01 11:22

I also said "user controllable parameters"

Chief taxi captain 2024-05-17 11:17

I suggest that 360 open source all its products, and then become the leading enterprise in the domestic open source industry through open source, leading everyone to compete with foreign enterprises

gamedot 2024-05-17 11:14

Old Zhou is deeply concerned about Huawei's great cause of open source. He is not a Huawei person, but has Huawei's soul.

Bright Stars 2 2024-05-31 23:28

Remove Unsafe? You don't want netty anymore?

CodeDoger 2024-05-02 20:48

35 It's too old to go to work and too early to retire at 60

zzeric 2024-04-28 20:01

Although France is the parent community, the core developers of OCCT on github are all Russians. Without Russians, the French parent community cannot continue to operate. So Huawei took over, moved to China, changed its name and resumed open source and community operations. What's the problem?

monkey_cici 2024-05-09 00:25

My I9 CPU, 64GB memory module and 3080Ti computer are inferior to the top configuration of 19999 on a tablet

MrChen89 2024-04-29 09:18

There are a group of people like this. I don't know what they have experienced. When it comes to HW, I can't say anything good, even if it's neutral

looly 2024-06-02 14:32

@Qingmiao Hutool has also been mentioned some loopholes that I think are relatively "low-level", or I think are not loopholes. At first, I was also very angry, but after thinking it through, I found that CVE's idea was that once you did not actively remind users that there was a pit, the user fell into the pit is your fault, that is, your vulnerability. For example, as a traffic policeman, you should remind everyone who crosses the road to pay attention to safety, and ask him to answer whether he knows. Once you don't remind someone and are hit by a car, you can't get away from it. Similarly, when using frameworks and tools, you should provide at least one parameter to remind users that there may be SQL injection vulnerabilities. Note that it is not in the comments, but in the method parameters, which is the user's responsibility. Therefore, it is not comprehensive to provide solutions in comments or documents.

Xiao Xu Middle aged 2024-06-01 06:49

thank

kangaroo 2024-06-01 22:23

The next version focuses on improving existing functions * improving internal power and qi * and continues to move towards the goal of Grand Master.

Ma Nong Little Fatty Brother 2024-05-16 14:40

I give you six seconds. I give you six moves with the same effect in the martial arts contest, which shows the invincibility and confidence of the master

-SORA- 2024-06-01 09:30

American characters

The seven in one little King Kong 2024-06-02 15:54

Those people only use resources, others are not developed by NPM...

All the way north GP 2024-04-25 14:55

America, the future of mankind

Xiao Xu Middle aged 2024-06-01 07:03

good

Happy LeapFrog 2024-05-18 09:18

But the question is: "What's the use of this for ordinary Android users?" Now the answer seems to be: "Almost nothing.".

jalena 2024-05-31 23:57

I can imagine that I will also receive the CVE repair request next week..... I don't use the key!!!!!!!!!

Single structure 2024-05-11 10:09

Selected as Open Source China's disgrace pillar

sweet potato chips 2024-05-31 22:08

Glue code consumes few resources

-SORA- 2024-04-30 17:07

When this happened in a foreign country, the comment area suddenly became very objective and rational**

haol666 2024-05-31 18:56

This story is powerful, I take it seriously, until I see the end.

sunday12345 2024-05-15 18:31

What does the bank do? It's blamed on the remote desktop. Persimmons really pick up soft pinches~?

Francesca 2024-05-19 18:00

Wine runs the Android emulator of Windows. Chrome OS is installed in the Android emulator. Linux environment is installed in chrome OS. Linux environment is installed in the Linux environment. Wine is installed in the Android emulator

Yoona520 2024-05-17 16:34

Zhou Hongyi is now living more and more like a clown. If he stays behind the scenes, he has to become an online celebrity. Can you learn from Lei Jun?

Shuimu Yi'an 2024-05-20 09:58

The news should be read continuously. I'm waiting for the third news besides rustdesk and teamviewer. Localized remote desktop software is far ahead.

osc_566335 2024-04-28 14:44

This is also called floor washing? Does it mean that Tesla will not wash the floor if it releases all the source code? Some people HWptds? That is to say, the language is ambiguous, which will also rise to the washing ground? Are some people too focused? Think the people he pays attention to must be staring at?

Li Yinghui 2024-05-09 16:40

Buddhism has a good word, evil opinion. In dealing with the world, it is meaningless to draw conclusions from preset positions; It is also important to receive good logic training.

Monkeys think of apes 2024-05-31 18:31

You can cheat your brother. Just don't cheat yourself

young crops 2024-06-01 16:21

There is no tipping point. There are also many official documents stating that SQL fragments involving direct string splicing need to be controlled by the user, and specific solutions are also provided. If you say that the value part is injected, then we are also 100% free of any dispute. This obvious SQL fragment is unrealistic for ORM to explain without your control, Since SQL allows splicing fragments, there must be some scenarios that cannot be forced into non SQL strings. It is also very simple. Have you ever thought about why not force them???

infoworld 2024-05-11 15:12

Universities should use open source free software instead of commercial ones. In this way, hands and feet will not be tied technically.

xiaoqibabby 2024-05-15 17:36

The bank is strongly required to be responsible for

osc_25732934 2024-06-01 19:30

It seems that the current version of the Foreign Function&Memory API is not as fast as that of jni, or even worse. In addition, before vallhala comes out, all interactions between java and c have to get an additional memory. Even if it comes out, it may not be possible to directly throw a copy of binary data into memory as a structure. When the two apis are completely stable, the day lily is cold

Xiao Xu Middle aged 2024-05-31 19:13

Very good

Small and beautiful software development 2024-06-01 05:06

Cheat one's job

Apizza 2024-06-01 17:52

You can switch from lodash to radash in 2024!!!

Brother Xiao Yang 2024-06-01 20:39

Isn't Ali developed? What are you afraid of? There's no need for every family to set up a set

GDWhisperer 2024-05-15 17:23

I transferred tens of thousands of yuan to my own account, which was under risk control. How did I do this? The bank should be responsible for this**

kakai 2024-05-10 10:21

The world only knows that Android was created by Google. Several people know that Android is only a product acquired by Google. Similarly, what is the problem with Huawei's contribution to the collection of OGG open source work and integration into its own proprietary product line?

oldpig 2024-04-28 09:59

”Huawei contributed all the source code "?, the title is completely inconsistent with the content.

Spark local mode operation

Hot content

Popular comments of the whole site

About the author

Author's Album

Author's other popular articles

Hot News

Hot software

OSCHINA Community

Online tools

Introduction

QQ group

Public account

Video number

Spark local mode operation

Hot content

Popular comments of the whole site

About the author

Author's Album

Author's other popular articles

Hot News

Recommended attention

Hot software

OSCHINA Community

Online tools

Introduction

QQ group

Public account

Video number