Python urllib.parse - Sogou test personal space - OSCHINA - Chinese open source technology exchange community

python-urllib.parse

2020/06/29 15:00

Reading number 68

preface

In the process of writing interface automation test cases recently, the editor needs to replace some parameters in the get request url with preset data, and replace the timeliness auth in the url with the return value of the auth generation method. After some research, we finally selected the parse module of python's urllib library.

The urllib.parse module provides a series of functions for manipulating URLs and their components. These functions are used for splitting or assembling.

Introduction to urllib.parse function

analysis:

1.ulrparse()

The return value of the function is a ParseResult object, which is similar to the tuple containing six elements.

 urllib_parse_urlparse.py from urllib.parse import urlparse url = ' http://test.dis.e.sogou/adlist?offset=0&auth=69CF80EA062863279B72612FA5443B6F&requestId=0025500016111592878436805&count=1&network=1 ' parsed = urlparse(url) print(parsed)

The six parts of the URL address that can be obtained by tuple indexing are: scheme, network location, path, path segment parameter (separated from the path by semicolon), query string and fragment.

 python3 urllib_parse_urlparse.py ParseResult(scheme='http',  netloc='test.dis.e.sogou', path='/adlist', params='', query='offset=0&auth=69CF80EA062863279B72612FA5443B6F&requestId=0025500016111592878436805&count=5&model=2&terminal=3&network=1', fragment='')

2.urlsplit()

The urlsplit() function can be used as an alternative to urlparse(), but it does not split the parameters in the URL.

Inverse analysis:

1.geturl ()

There is more than one way to get a complete URL string by reassembling the parts of the split URL. The parsed URL object has a geturl () method.

 urllib_parse_geturl.py
 from urllib.parse import urlparse
 original = ' http://test.dis.e.sogou/adlist?offset=0&auth=69CF80EA062863279B72612FA5443B6F&requestId=0025500016111592878436805&count=5&model=2&terminal=3&network=1 '  print('ORIG  :', original)  parsed = parse.urlparse(original)  print('PARSED:', parsed.geturl())

 $ python3 urllib_parse_geturl.py ORIG  :  http://test.dis.e.sogou/adlist?offset=0&auth=69CF80EA062863279B72612FA5443B6F&requestId=0025500016111592878436805&count=5&model=2&terminal=3&network=1 PARSED:  http://test.dis.e.sogou/adlist?offset=0&auth=69CF80EA062863279B72612FA5443B6F&requestId=0025500016111592878436805&count=5&model=2&terminal=3&network=1

Geturl () is only valid for objects returned by urlparse() or urlsplit().

2.ulrunparse()

You can use urlunparse () to assemble a regular string tuple into a URL address.

Splicing:

1.urljoin()

In addition to the urlparse() function used to parse URLs, the urllib.parse module also contains the urljoin() function, which can be used to create absolute URLs from fragments of relative addresses.

 urllib_parse_urljoin.py
 from urllib.parse import urljoin print(urljoin(' http://www.example.com/path/file.html ',               'anotherfile.html')) print(urljoin(' http://www.example.com/path/file.html ',               '../ anotherfile.html'))

In this example, when splicing the second URL, the ("../") representing the relative path is taken into account.

 $ python3 urllib_parse_urljoin.py http://www.example.com/path/anotherfile.html http://www.example.com/anotherfile.html

Non relative paths are handled in the same way as os. path. join().

 urllib_parse_urljoin_with_path.py
 print(urljoin(' http://www.example.com/path/ ',               '/subpath/file.html')) print(urljoin(' http://www.example.com/path/ ',               'subpath/file.html'))

If the path to be spliced to the URL address starts with a slash (/), the URL address will be reset at the top level with that path. Otherwise, it is only added to the end of the URL path

 $ python3 urllib_parse_urljoin_with_path.py
 http://www.example.com/subpath/file.html http://www.example.com/path/subpath/file.html

Code Query Parameters ：

1.ulrencode()

Query parameters must be encoded before adding URL addresses

 urllib_parse_urlencode.py
 from urllib.parse import urlencode query_args = {     'q': 'query string',     'foo': 'bar', } encoded_args = urlencode(query_args) print('Encoded:', encoded_args)

The encoding process will replace some special characters, such as spaces, to ensure that the format of the query string passed to the server is standard.

 $ python3 urllib_parse_urlencode.py
 Encoded: q=query+string&foo=bar

In the query string, you can set doseq to True when calling urlencode() in order to make each of a sequence of variable values appear in a separate way.

2.parse_qs()

The result returned by parse_qs() is a dictionary. Each item in the dictionary is a list of query names and their corresponding (one or more) values, while parse_qsl() returns a list of tuples. Each tuple is a pair of query names and query values

 $ python3 urllib_parse_parse_qs.py
 parse_qs : {'foo': ['foo1', 'foo2']} parse_qsl: [('foo', 'foo1'), ('foo', 'foo2')]

The use of ulllib.parse in the framework

 test_dippatcher_adlist.py
 url= ' http://test.dis.e.sogou/adlist?offset=0&auth=69CF80EA062863279B72612FA5443B6F&requestId=0025500016111592878436805&count=3&model=ios&terminal=1&version=2&network=1 ' http: //test.dis.e.sogou/adlist? offset=0&auth=69CF80EA062863279B72612FA5443B6F&requestId=0025500016111592878436805&count=3&model=ios&terminal=1&version=2&network=1' #Get request_id and auth request_id = generate_requestId( expect [ 'platformId' ],  expect [ 'posId' ]) auth = generate_auth(request_id,  expect [ 'token' ]) #Modify the parameters in Url and replace request_id and auth #Analysis URL url_parsed = parse.urlparse(url) bits = list(url_parsed) qs = parse.parse_qs(bits[ four ]) #Replace interface input parameters in qs qs[ 'requestId' ] = request_id qs[ 'auth' ] = auth qs[ 'offset' ] =  expect [ 'offset' ] qs[ 'count' ] =  expect [ 'count' ] qs[ 'model' ] =  expect [ 'model' ] qs[ 'terminal' ] =  expect [ 'terminal' ] qs[ 'version' ] =  expect [ 'version' ] qs[ 'network' ] =  expect [ 'network' ] #Edit Query Parameters bits[ four ] = parse.urlencode(qs) #URL reverse resolution url_new = parse.urlunparse(bits) print(url_new)

For better understanding, output the results of each part.

 $ python3 test_dispatcher_adlist.py
 bits: [ 'http' , 'test.dis.e.sogou' , '/adlist' , '' , "offset=0&auth=69CF80EA062863279B72612FA5443B6F&requestId=0025500016111592878436805&count=3&model=ios&terminal=1&version=2&network=1' http://test.dis.e.sogou/adlist?offset=0&auth=69CF80EA062863279B72612FA5443B6F&requestId=0025500016111592878436805&count=3&model=ios&terminal=1&version=2&network=1 " , '' ] qs: { 'offset' : [ '0' ], 'auth' : [ '69CF80EA062863279B72612FA5443B6F' , '69CF80EA062863279B72612FA5443B6F' ], 'requestId' : [ '0025500016111592878436805' , '0025500016111592878436805' ], 'count' : [ '3' , '3' ], 'model' : [ 'ios' , 'ios' ], 'terminal' : [ '1' , '1' ], 'version' : [ '2' , '2' ], 'network' : [ "1' http://test.dis.e.sogou/adlist?offset=0 " , '1' ]} bits[ four ]: offset= zero &auth= eight thousand two hundred and fifteen f55af287a62a29efe7a70fd3ba0d&requestId= 0025500016111593405114583 &count= one &model=eee&terminal= one &version=eee&network= one http: //test.dis.e.sogou/adlist? offset=0&auth=8215f55af287a62a29efe7a70fd3ba0d&requestId=0025500016111593405114583&count=1&model=eee&terminal=1&version=eee&network=1

Sogou test WeChat signal: Qa_xiaoming

Sogou test QQ fan group: 459645679

This article is shared from the WeChat official account Sogou QA.
In case of infringement, please contact support@oschina.cn Delete.
Participation in this article“ OSC Source Innovation Plan ”, welcome you to join us and share with us.

gamedot 2024-05-17 11:14

Old Zhou is deeply concerned about Huawei's great cause of open source. He is not a Huawei person, but has Huawei's soul.

zzeric 2024-04-28 20:01

Although France is the parent community, the core developers of OCCT on github are all Russians. Without Russians, the French parent community cannot continue to operate. So Huawei took over, moved to China, changed its name and resumed open source and community operations. What's the problem?

Ning Jinnong 2024-06-01 21:04

Correct it. The example of loading the library is wrong. It should be # library=@ loading the dynamic library, "./yards to the treasurer. dll"

Shuimu Yi'an 2024-05-20 09:58

The news should be read continuously. I'm waiting for the third news besides rustdesk and teamviewer. Localized remote desktop software is far ahead.

Ma Nong Little Fatty Brother 2024-05-16 14:40

I give you six seconds. I give you six moves with the same effect in the martial arts contest, which shows the invincibility and confidence of the master

Happy LeapFrog 2024-05-18 09:18

But the question is: "What's the use of this for ordinary Android users?" Now the answer seems to be: "Almost nothing.".

kangaroo 2024-06-01 22:23

The next version focuses on improving existing functions * improving internal power and qi * and continues to move towards the goal of Grand Master.

hanf 2024-05-31 17:45

If only the design and architecture are similar, what's the point? Good things must be learned, and you can't prove that the design is not the same. As for the source code, you also said that neither Oracle nor Damon is open source, and you can't prove it. There are many people who question Dream, but so far, no one has come up with strong evidence. You should at least provide evidence to copy

-SORA- 2024-06-01 09:30

American characters

looly 2024-06-02 14:32

@Qingmiao Hutool has also been mentioned some loopholes that I think are relatively "low-level", or I think are not loopholes. At first, I was also very angry, but after thinking it through, I found that CVE's idea was that once you did not actively remind users that there was a pit, the user fell into the pit is your fault, that is, your vulnerability. For example, as a traffic policeman, you should remind everyone who crosses the road to pay attention to safety, and ask him to answer whether he knows. Once you don't remind someone and are hit by a car, you can't get away from it. Similarly, when using frameworks and tools, you should provide at least one parameter to remind users that there may be SQL injection vulnerabilities. Note that it is not in the comments, but in the method parameters, which is the user's responsibility. Therefore, it is not comprehensive to provide solutions in comments or documents.

Love to eat raw pears 2024-06-01 11:48

Why is this so-called "vulnerability" not a vulnerability? Spring, MyBatis and other frameworks can accept all kinds of CVE criticism, while MyBatisPlus has to dump the pot and accuse programmers of being too low-level# There is a difference. The premise is that you write XML, MyBatisPlus encapsulates Wrapper and claims to simplify code. Since it encapsulates and hides $#, it is not appropriate to do some necessary security checks? Instead of doubting the authority of CVE, you should know that SQL ->MyBatis ->MyBatisPlus ->various back-end scaffolds have multiple layers, each layer is simplifying, and each layer is throwing away the upper layer of the boiler. Who dares to use them. The programmers who use MyBatisPlus can't be expected to be at a high level. Every programmer wants to save effort. The front-end parameters can be directly obtained by HttpServletRequest from the back-end. Wrapper splicing can be found everywhere. If something goes wrong, is it the front-end or the framework? According to Qingmiao, can the injection vulnerability of the previous log4j and the deletion vulnerability of the Druid be used to eliminate low-level programmers?

Rocket ship 2024-05-31 19:22

It's a ghost anyway.

osc_25732934 2024-06-01 19:30

It seems that the current version of the Foreign Function&Memory API is not as fast as that of jni, or even worse. In addition, before vallhala comes out, all interactions between java and c have to get an additional memory. Even if it comes out, it may not be possible to directly throw a copy of binary data into memory as a structure. When the two apis are completely stable, the day lily is cold

CodeDoger 2024-05-02 20:48

35 It's too old to go to work and too early to retire at 60

-SORA- 2024-04-30 17:07

When this happened in a foreign country, the comment area suddenly became very objective and rational**

zhy 2024-05-16 13:16

At the end of Shannon is Nong

Single structure 2024-05-11 10:09

Selected as Open Source China's disgrace pillar

sweet potato chips 2024-05-31 22:08

Glue code consumes few resources

Brother Xiao Yang 2024-06-01 20:39

Isn't Ali developed? What are you afraid of? There's no need for every family to set up a set

Voice of God 2024-06-01 20:47

By default, injection ($) and splicing are turned off. If you want to use it, you need to sign the birth and death form and press the fingerprint.

Xiao Xu Middle aged 2024-06-01 07:03

good

Apizza 2024-06-01 17:52

You can switch from lodash to radash in 2024!!!

Chief taxi captain 2024-05-17 11:17

I suggest that 360 open source all its products, and then become the leading enterprise in the domestic open source industry through open source, leading everyone to compete with foreign enterprises

Starry Night Destiny 2024-06-01 21:49

It feels like Mybatis. It's OK to provide users with optional security solutions. It's useless for users to complain about this problem

Yoona520 2024-05-17 16:34

Zhou Hongyi is now living more and more like a clown. If he stays behind the scenes, he has to become an online celebrity. Can you learn from Lei Jun?

Monkeys think of apes 2024-05-31 18:31

You can cheat your brother. Just don't cheat yourself

monkey_cici 2024-05-09 00:25

My I9 CPU, 64GB memory module and 3080Ti computer are inferior to the top configuration of 19999 on a tablet

Small and beautiful software development 2024-06-01 05:06

Cheat one's job

Code craftsman 2024-06-01 11:22

I also said "user controllable parameters"

Shen Lang Panda 2024-06-01 08:16

You can directly ask questions in the project work order. The comment area is not suitable for answering such questions

Francesca 2024-05-19 18:00

Wine runs the Android emulator of Windows. Chrome OS is installed in the Android emulator. Linux environment is installed in chrome OS. Linux environment is installed in the Linux environment. Wine is installed in the Android emulator

Xiao Xu Middle aged 2024-05-31 19:13

Very good

Dogo_Little People 2024-06-02 12:24

Not everyone will go to see the document in full detail. As a general basic framework, the method naming should consider not only readability but also understandability. At least, it should also establish a cognition for developers. LambdaQueryWrapper is recommended. The official only briefly said that QueryWrapper may lead to SQL injection risks, There are no detailed examples (many people don't understand what SQL injection is). Now I met a jerk and submitted it to CVE to see who is the most powerful

infoworld 2024-05-11 15:12

Universities should use open source free software instead of commercial ones. In this way, hands and feet will not be tied technically.

MrChen89 2024-04-29 09:18

There are a group of people like this. I don't know what they have experienced. When it comes to HW, I can't say anything good, even if it's neutral

GDWhisperer 2024-05-15 17:23

I transferred tens of thousands of yuan to my own account, which was under risk control. How did I do this? The bank should be responsible for this**

xiaoqibabby 2024-05-15 17:36

The bank is strongly required to be responsible for

sunday12345 2024-05-15 18:31

What does the bank do? It's blamed on the remote desktop. Persimmons really pick up soft pinches~?

Hakuna 2024-05-31 18:28

It is compatible with Oracle, but does not know "just" or "just". Those who can be compatible with Oracle and do well are real men and real warriors. You should know that compatibility means that even bugs must be compatible, and you have no other code that can not be copied. It's all based on real skills and understanding of oracle.

osc_92224065 2024-04-29 10:57

Long term oppressed outsourcing of state-owned enterprises

One code Yma 2024-05-09 09:58

Recently, I often go to interviews. People who hate Ali background most regard me as a fool, even though I am a fool

Xiao Xu Middle aged 2024-06-01 06:49

thank

One code Yma 2024-05-06 09:14

My technical article was moved by CSDN. Why didn't anyone step on the sewing machine? This kind of report is a joke to me. The monsters with background are fine, and the monsters without background fight to death

Li Yinghui 2024-05-09 16:40

Buddhism has a good word, evil opinion. In dealing with the world, it is meaningless to draw conclusions from preset positions; It is also important to receive good logic training.

kakai 2024-05-10 10:21

The world only knows that Android was created by Google. Several people know that Android is only a product acquired by Google. Similarly, what is the problem with Huawei's contribution to the collection of OGG open source work and integration into its own proprietary product line?

haol666 2024-05-31 18:56

This story is powerful, I take it seriously, until I see the end.

Bright 2024-05-19 23:25

What a fool! I killed myself. How can people deal with me later.

Yeah, for 2024-05-17 13:42

That's too right. Old Zhou can't control Google, but he can control 360. Do not do to others what you do not want. All 360 products should be opened first.

oldpig 2024-04-28 09:59

”Huawei contributed all the source code "?, the title is completely inconsistent with the content.

People are addicted to food 2024-06-01 13:53

History history combination

Bright Stars 2 2024-05-31 23:28

Remove Unsafe? You don't want netty anymore?

The seven in one little King Kong 2024-06-02 15:54

Those people only use resources, others are not developed by NPM...

All the way north GP 2024-04-25 14:55

America, the future of mankind

Love to eat raw pears 2024-06-01 19:18

Don't expect programmers to have a deep understanding of the document. I still think that since the tool hides the details of $#, some necessary security checks are necessary. Many people do not use MybatisPlus directly, but use various so-called rapid development platforms. The MyBatisPlus rapid development platform Snowy, Guns, etc., has an impression that many versions have the problem of using Wrapper directly to splice the Request parameter. I remember that JeecgBoot was opened a lot of CVEs last year or the year before last because of the Wrapper splicing problem. Do you know the author of ibeetl? Many CVE blaming holes have been opened before. The problem is similar. The lack of basic knowledge "script editing permission" is actively handed over to the front end. What a low-level error or even low-energy behavior. However, I accepted it with an open mind and added a white list check.

osc_566335 2024-04-28 14:44

This is also called floor washing? Does it mean that Tesla will not wash the floor if it releases all the source code? Some people HWptds? That is to say, the language is ambiguous, which will also rise to the washing ground? Are some people too focused? Think the people he pays attention to must be staring at?

Qin Liming 2024-05-11 09:12

be devoid of any sense of shame

Yokesily 2024-06-02 15:11

So designed

young crops 2024-06-01 16:21

There is no tipping point. There are also many official documents stating that SQL fragments involving direct string splicing need to be controlled by the user, and specific solutions are also provided. If you say that the value part is injected, then we are also 100% free of any dispute. This obvious SQL fragment is unrealistic for ORM to explain without your control, Since SQL allows splicing fragments, there must be some scenarios that cannot be forced into non SQL strings. It is also very simple. Have you ever thought about why not force them???

jalena 2024-05-31 23:57

I can imagine that I will also receive the CVE repair request next week..... I don't use the key!!!!!!!!!

python-urllib.parse

Hot content

Popular comments of the whole site

About the author

Author's Album

Author's other popular articles

Hot News

Hot software

OSCHINA Community

Online tools

Introduction

QQ group

Public account

Video number

python-urllib.parse

Hot content

Popular comments of the whole site

About the author

Author's Album

Author's other popular articles

Hot News

Recommended attention

Hot software

OSCHINA Community

Online tools

Introduction

QQ group

Public account

Video number